PDA

View Full Version : DomU's seethe same NSS volume, but not same data



jmcg
19-Sep-2013, 17:14
I had posted this on the oes forum but think it is more applicable to this thread. Here is the situation:

I have two oes11 sp1 servers virtualized on a xen sles 11 sp 2 server. The host has a raid 5 from which an nss volume is created from. Both servers see the same volume in the same path in their directory structure. Yet, the two servers do not see the same directories on the volume. I know this is not good but am hoping that someone can give me some pointers if my plan is not the best.

More specifics:

Server 1
Examine the /dev/disk/by-path I see the xvdb which is the nss volume. That is fine. When I examine the output from df -h command it shows up as expected.

Server 2
Examine the /dev/disk/by-path I see the xvdb which is not shown in the output from df -h. I also see the xvdd in both the /dev/disk/by-path as is expected. Yet the xvdb is a path that can be drilled down into from the terminal or the gui.

The bad part is this xvdb has the directory for groupwise gwia. My plan tonight is to stop gw and to copy it from the xvdb seen by server 2 to the xvdd on server 2. Change the path in C1 for gwia, start it back up, that should take care of that, but....

How do I detach (is that the right syntax) the nss vol xvdb from server2?

Also, now server 2, which runs our Groupwise system,which seems to be running fine, has become sluggish. I can connect to it via ssh but the terminal screen never becomes responsive. On the host terminal, using the console of the DomU I couldn't get top to be displayed on the terminal so I issued a init 3 and took it out of init 5, then tried to go back to init 5 and now all I see is a a spinning progress icon but the process doesn't finish to come back to the desktop.

Even the Dom0 is becoming less responsive. I try to examine disk space by df -h and the process won't finish and display the data. I am hoping that someone will offer some suggestions before this whole system goes down.

KBOYLE
19-Sep-2013, 19:22
jmcg wrote:

>
> I had posted this on the oes forum but think it is more applicable to
> this thread. Here is the situation:
>
> I have two oes11 sp1 servers virtualized on a xen sles 11 sp 2 server.

Okay...

> The host has a raid 5 from which an nss volume is created from.

Is this hardware RAID device? Do you use the whole array or have you
created a separate Logical Volume (LUN)? Or, are you using Linux
software RAID?

In any case, your Dom0 should see a disk (sdX) if you are using
hardware RAID or an mdX device if you are using Linux software RAID.
Which do you have?

Now, SLES11-SP2 doesn't know anything about NSS so I assume you are
passing the sdX/mdX device to each of your two OES DomUs? And that in
one of your DomUs you created an NSS volume?

The first thing to verify is that you got this part of it right. There
are some special consideration when you share a disk.

Have you seen TID 7004451?
How to share a XEN disk with two or more virtual machines.
http://www.novell.com/support/kb/doc.php?id=7004451

It says, in part:
> Customer needs to share a disk file (i.e.
> /var/lib/xen/images/VM-NAME/disk0) or a physical device (i.e.
> /dev/sdb1) with two or more XEN virtual machines in order to be able
> to install a cluster services like Linux Pacemaker or Novell Cluster
> Services.

All this does is ensure that a device can be shared between multiple
DomUs.

> Both servers see the same volume in the same path in their
> directory structure.

This is what you would expect but things get complicated if both
servers have write access. Normally you can't have two separate systems
update the same filesystem. Some filesystems do support this shared
access, others do not.

This is not really a virtualization issue. As I suggested in your other
post:

> you should post this one in 'OES-L: Storage and Backup'
>
(https://forums.novell.com/novell-product-discussions/file-networking-services/open-enterprise-server/oes-linux/oes-l-storage-backup/)
> where you'll find those with the necessary expertise to help.


> Yet, the two servers do not see the same directories on the
> volume. I know this is not good but am hoping that someone can give me
> some pointers if my plan is not the best.
>
> More specifics:
>
> Server 1
> Examine the /dev/disk/by-path I see the xvdb which is the nss volume.
> That is fine. When I examine the output from df -h command it shows up
> as expected.
>
> Server 2
> Examine the /dev/disk/by-path I see the xvdb which is not shown in the
> output from df -h. I also see the xvdd in both the /dev/disk/by-path
> as is expected. Yet the xvdb is a path that can be drilled down into
> from the terminal or the gui.

A shared disk employs a locking mechanism to prevent concurrent updates
from multiple systems. If a system thinks it has full access to a
device but doesn't, lots of weird things can happen... including data
corruption.


> The bad part is this xvdb has the directory for groupwise gwia. My
> plan tonight is to stop gw and to copy it from the xvdb seen by
> server 2 to the xvdd on server 2. Change the path in C1 for gwia,
> start it back up, that should take care of that, but....

I really don't know what xvdb or xvdd are used for but that is beside
the point. If you are not careful you may very well wipe out *all* your
data.

Rather than deal with the issues in the rest of your post, I suggest
you refer to the documentation for some guidelines then post any
follow-up questions in the 'OES-L: Storage and Backup' forum.

OES 11 SP1: File System Management Guide
http://www.novell.com/documentation/oes11/stor_filesys_lx/data/hn0r5fzo.html#hn0r5fzo

OES 11 SP1: NSS File System Administration Guide for Linux
http://www.novell.com/documentation/oes11/stor_nss_lx/data/front.html#front


--
Kevin Boyle - Knowledge Partner
If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below...

jmcg
19-Sep-2013, 22:26
Kevin,
I won't bore you with my details, but the bottom line is that the two servers do not see the same disk. The names were the same to the first part of the directory structure, but after that they are different.

To give you more specifics, the xen host has a RAID 1 that the xen os is installed on and has the OS of each DomU on it. The host also has a RAID 5 on it that I have carved up into different partitions for each DomU to use. The system is on its 6 th year and was my first experience in virtualization. The xvda, xvdb, xvdc, and xvdd are the different partitions of the RAID 5 that different DomU's have assigned to them. So all is good on the storage end.

However, I was having some performance issues as mentioned at the end of my first post. I could not get the host to display df -h on the terminal. I wasn't sure how to troubleshoot why that was occurring, I shut down the DomU's one at a time to see if I could get the command to display, but it wouldn't. So I eventually shutdown all the DomU's and restart the Xen server and all seems to be fine. I think my problem is that the RAID one is 86% allocated. There is free space for each of the DomU's on the disk images, but to the host it only has 14% space left. Won't that cause performance issues? Isn't that problematic or potentially so? Thank you for the help.

KBOYLE
20-Sep-2013, 01:00
jmcg wrote:

>
> Kevin,
> I won't bore you with my details, but the bottom line is that the two
> servers do not see the same disk.

That's good to hear but it is not what I understood when I read:

> The host has a raid 5 from which an nss volume is created from. Both
> servers see the same volume in the same path in their directory
> structure.

> The names were the same to the first part of the directory structure,
> but after that they are different.

I don't understand what that means.


> To give you more specifics, the xen host has a RAID 1 that the xen os
> is installed on and has the OS of each DomU on it. The host also has
> a RAID 5 on it that I have carved up into different partitions for
> each DomU to use. The system is on its 6 th year and was my first
> experience in virtualization.

So far so good.


> The xvda, xvdb, xvdc, and xvdd are the
> different partitions of the RAID 5 that different DomU's have
> assigned to them. So all is good on the storage end.

Again, I'm confused...

When you say "The host also has a RAID 5 on it that I have carved up
into different partitions", that suggests to me that your Dom0 sees a
single large drive. If you partitioned it, you would have sdd1, sdd2,
sdd3, etc.

The other possibility is that you used your RAID utility to create
Logical Volumes (LUNs) in which case Dom0 would see separate drives but
I would expect them to be seen in Dom0 as sda, sdb, sdc, etc. Your DomU
would see the device as xvdX but regardless how the device in known in
Dom0, it would likely have the same name in each DomU (e.g. xvda). So
where do the names xvda, xvdb, xvdc, and xvdd come from?


> However, I was having some performance issues as mentioned at the end
> of my first post. I could not get the host to display df -h on the
> terminal. I wasn't sure how to troubleshoot why that was occurring, I
> shut down the DomU's one at a time to see if I could get the command
> to display, but it wouldn't. So I eventually shutdown all the DomU's
> and restart the Xen server and all seems to be fine.

Okay.


> I think my problem is that the RAID one is 86% allocated.

If you are explaining it correctly, that is not necessarily a problem.
For example, if you had a 100 GB RAID 1 array and created three
partitions (LUNs) each of which was 25 GB, your array would be 75%
allocated, and that is fine, but somehow I don't think that is what you
are saying.

> There is free space for each of the DomU's on the disk images, but to
the host it only has 14% space left.

Again, you haven't actually described how you have configured your RAID
1 array. When you say "There is free space for each of the DomU's on
the disk images" am I to assume that you are using file backed (which
means that the virtual disk is a single image file on a larger physical
disk) storage for your DomU's system disk? If that is what you have
done, are you using a sparse image file, which means that the virtual
disk is a single image file, but the space is not preallocated?

If you are using a 50 GB sparse image file, the DomU would see a 50 GB
disk but the file size on your Dom0 may only be 30 GB. The Dom0 file
will grow to 50 GB as the DomU uses more storage space.

*If* this is how your system is configured, and *if* your RAID 1 array
is seen by your Dom0 as a single drive/partition (other than swap), and
*if* you only have 14% free space, then *yes*, this is a problem!


> Won't that cause performance issues?

Sparse files by themselves can be a performance issue but you haven't
confirmed that's what you're using.


> Isn't that problematic or potentially so?

Very much so. Very bad things can happen to your system if you run out
of space. Depending upon how you have configured storage your Dom0, you
may exhaust all available storage at a specific mount point and still
show free space available on the drive.

Much of what I have said is speculation based on assumptions. Have I
misunderstood (again) what you are trying to explain?

--
Kevin Boyle - Knowledge Partner
If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below...

jmcg
24-Sep-2013, 16:26
I hope that my ignorance is not too obvious to you or anyone else who views this, but the more I put into this, the better I understand my system.
First, the physical server has a RAID 1 Ė hardware type and a hardware type RAID 5. Those are physical discs. The host OS is SLES 11 and is install as a Xen server, minimal extras, on the RAID 1 .
The RAID 1 shows up in the terminal (df Ėh) as c0d0p1 in the /dev/cciss/ directory. That is the disk space that is 78% used up.

Each DomU (guests or vmís) are installed as disk files on the RAID 1. That is, not only is the Host OS installed on the RAID 1 but also the disk files for each of the DomUís (they are in the /var/lib/xen/images/ directory) are too, but they are technically files (disk images).

The RAID 5 shows up in the host directory (/dev/cciss/c0d1x) with each of the partitions (linux) that I have created for the DomUís to use. Each DomU has a partition of the RAID 5 attached to it. They show up as /dev/cciss/c0d1p2 or 3 or 4, depending on the order I created them.

Within each of the domUís, I see those partitions (RAID 5) as xvdb or xvdd or whatever. That is what I see from the terminal of the domU when I enter df Ėh. I donít see them from the Dom0ís terminal because they are not attached to it, only to the DomUís.

So, it seems that what I have done is used disk files for the OS on each DomU but am using actually using partitions on the RAID 5 for each of the DomUís. So I am mixing and matching. Is that bad?

Now with all that said I have one more issue. One of the DomUís that is an OES11sp1 server that is running my primary domain for groupwise has become sluggish. Over a period of 5 or so days it becomes less responsive. Now I canít even get top or df Ėh to activate from the terminal or from an ssh session. The groupwise system is running fine, but the server seems busy. What can I do to investigate why it is not responsive to terminal commands? Yesterday I did a restart of the groupwise system and it took 5 or 6 minutes to restart and it only has 100 users, a small system. I appreciate your help.

KBOYLE
25-Sep-2013, 01:51
jmcg wrote:

>
> I hope that my ignorance is not too obvious to you or anyone else who
> views this, but the more I put into this, the better I understand my
> system.

That is a good thing. Detailed questions make us think and even
question what has been put into place to help us better understand and
better able to identify issues.



> First, the physical server has a RAID 1 hardware type and a hardware
> type RAID 5. Those are physical discs. The host OS is SLES 11 and is
> install as a Xen server, minimal extras, on the RAID 1 .
> The RAID 1 shows up in the terminal (df h) as c0d0p1 in the
> /dev/cciss/ directory. That is the disk space that is 78% used up.

Hardware RAID is good. 78% used (or is it 86% as stated in your other
post?) doesn't help much without knowing a bit more. Can you provide
the output from these commands run on Dom0? It will explain a lot more
than I can get by asking specific questions.



fdisk -l
cat /etc/fstab


Note: That's a lower case ELL on the fdisk command.

Please post your results between "code" tags (using the "#" from the
web interface) to make the output more readable.



> Each DomU (guests or vms) are installed as disk files on the RAID 1.
> That is, not only is the Host OS installed on the RAID 1 but also the
> disk files for each of the DomUs (they are in the /var/lib/xen/images/
> directory) are too, but they are technically files (disk images).

So far, that's fine. The commands I asked you to run will tell us more.



> The RAID 5 shows up in the host directory (/dev/cciss/c0d1x) with each
> of the partitions (linux) that I have created for the DomUs to use.
> Each DomU has a partition of the RAID 5 attached to it. They show up
> as /dev/cciss/c0d1p2 or 3 or 4, depending on the order I created
> them.

It is good that you separated your data storage from that which
contains your operating system.



> Within each of the domUs, I see those partitions (RAID 5) as xvdb or
> xvdd or whatever. That is what I see from the terminal of the domU
> when I enter df h. I dont see them from the Dom0s terminal because
> they are not attached to it, only to the DomUs.

What you say makes sense but if on a particular DomU you can see
/dev/xvdd, that suggests to me you have assigned one file-backed
storage device for the operating system and three separate partitions
from your RAID 5 device. Is that correct? Why three devices for data?
(Just curious.)



> So, it seems that what I have done is used disk files for the OS on
> each DomU but am using actually using partitions on the RAID 5 for
> each of the DomUs. So I am mixing and matching. Is that bad?

No, that is not a bad idea depending upon what you are trying to
achieve. File-backed storage is created by default. It offers some
flexibilities and the ability to "thin provision" but is not
recommended for use with NSS data. Partitions on the RAID 5 (a block
storage device) offer better performance and are recommended for NSS.


> Now with all that said I have one more issue. One of the DomUs that
> is an OES11sp1 server that is running my primary domain for groupwise
> has become sluggish. Over a period of 5 or so days it becomes less
> responsive. Now I cant even get top or df h to activate from the
> terminal or from an ssh session. The groupwise system is running
> fine, but the server seems busy. What can I do to investigate why it
> is not responsive to terminal commands? Yesterday I did a restart of
> the groupwise system and it took 5 or 6 minutes to restart and it
> only has 100 users, a small system. I appreciate your help.

The first thing we have to determine is whether this issue lies within
your Dom0 or your DomU. If only one DomU is affected, I would suspect
the DomU (Server 2?).

Please run these commands on your DomU:


fdisk -l
cat /etc/fstab


If you shutdown GroupWise, is your server more responsive?

Did you move GWIA as you planned to do in your OP?

Poor performance can be caused by IO bottlenecks. Have you checked for
excessive swapping on each server or verified that there are no disk
errors that are causing excessive retries?

One other suggestion: If this is a production server and you need to
get this resolved quickly, have you considered opening a Service
Request to get some hands-on support?

Please provide the requested information to get a better understanding
of what's happening.

--
Kevin Boyle - Knowledge Partner
If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below...

jmcg
25-Sep-2013, 16:14
[QUOTE=KBOYLE;16512]jmcg wrote:

>
> I hope that my ignorance is not too obvious to you or anyone else who
> views this, but the more I put into this, the better I understand my
> system.

That is a good thing. Detailed questions make us think and even
question what has been put into place to help us better understand and
better able to identify issues.




> First, the physical server has a RAID 1 hardware type and a hardware
> type RAID 5. Those are physical discs. The host OS is SLES 11 and is
> install as a Xen server, minimal extras, on the RAID 1 .
> The RAID 1 shows up in the terminal (df h) as c0d0p1 in the
> /dev/cciss/ directory. That is the disk space that is 78% used up.

Hardware RAID is good. 78% used (or is it 86% as stated in your other
post?) doesn't help much without knowing a bit more. Can you provide
the output from these commands run on Dom0? It will explain a lot more
than I can get by asking specific questions.



fdisk -l
cat /etc/fstab


Note: That's a lower case ELL on the fdisk command.

Please post your results between "code" tags (using the "#" from the
web interface) to make the output more readable.


[CODE]
SLES-XEN:/ # fdisk -l

Disk /dev/cciss/c0d0: 146.8 GB, 146778685440 bytes
255 heads, 63 sectors/track, 17844 cylinders, total 286677120 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00075c47

Device Boot Start End Blocks Id System
/dev/cciss/c0d0p1 63 4209029 2104483+ 82 Linux swap / Solaris
/dev/cciss/c0d0p2 * 4209030 286663859 141227415 83 Linux

Disk /dev/cciss/c0d1: 899.9 GB, 899898718208 bytes
255 heads, 63 sectors/track, 109406 cylinders, total 1757614684 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0006a452

Device Boot Start End Blocks Id System
/dev/cciss/c0d1p1 32 706876064 353438016+ 5 Extended
/dev/cciss/c0d1p2 706876065 1757607389 525365662+ 83 Linux
/dev/cciss/c0d1p5 64 195309599 97654768 83 Linux
/dev/cciss/c0d1p6 195309632 292968479 48829424 83 Linux
/dev/cciss/c0d1p7 292977468 481966064 94494298+ 83 Linux
SLES-XEN:/ # cat /etc/fstab
/dev/disk/by-id/cciss-3600508b1001032373120202020200004-part2 / reiserfs acl,user_xattr 1 1
/dev/disk/by-id/cciss-3600508b1001032373120202020200004-part1 swap swap defaults 0 0
proc /proc proc defaults 0 0
sysfs /sys sysfs noauto 0 0
debugfs /sys/kernel/debug debugfs noauto 0 0
usbfs /proc/bus/usb usbfs noauto 0 0
devpts /dev/pts devpts mode=0620,gid=5 0 0
/dev/disk/by-id/cciss-3600508b1001032373120202020200005-part4 /mnt/mount ext3 rw 0 0
##/dev/cciss/c0d1p4 /mnt/mount ext3 rw 0 0
##next line edited by jmcg 21May2012 so partition not mounted
##/dev/disk/by-id/cciss-3600508b1001032373120202020200005-part3 /mnt/mount2 ext3 acl,user_xattr 1 2
/dev/disk/by-id/cciss-3600508b1001032373120202020200005-part6 /mnt/mount3 ext3 acl,user_xattr 1 2
SLES-XEN:/ #

[CODE]



> Each DomU (guests or vms) are installed as disk files on the RAID 1.
> That is, not only is the Host OS installed on the RAID 1 but also the
> disk files for each of the DomUs (they are in the /var/lib/xen/images/
> directory) are too, but they are technically files (disk images).

So far, that's fine. The commands I asked you to run will tell us more.



> The RAID 5 shows up in the host directory (/dev/cciss/c0d1x) with each
> of the partitions (linux) that I have created for the DomUs to use.
> Each DomU has a partition of the RAID 5 attached to it. They show up
> as /dev/cciss/c0d1p2 or 3 or 4, depending on the order I created
> them.

It is good that you separated your data storage from that which
contains your operating system.



> Within each of the domUs, I see those partitions (RAID 5) as xvdb or
> xvdd or whatever. That is what I see from the terminal of the domU
> when I enter df h. I dont see them from the Dom0s terminal because
> they are not attached to it, only to the DomUs.

What you say makes sense but if on a particular DomU you can see
/dev/xvdd, that suggests to me you have assigned one file-backed
storage device for the operating system and three separate partitions
from your RAID 5 device. Is that correct? Why three devices for data?
(Just curious.)



> So, it seems that what I have done is used disk files for the OS on
> each DomU but am using actually using partitions on the RAID 5 for
> each of the DomUs. So I am mixing and matching. Is that bad?

No, that is not a bad idea depending upon what you are trying to
achieve. File-backed storage is created by default. It offers some
flexibilities and the ability to "thin provision" but is not
recommended for use with NSS data. Partitions on the RAID 5 (a block
storage device) offer better performance and are recommended for NSS.


> Now with all that said I have one more issue. One of the DomUs that
> is an OES11sp1 server that is running my primary domain for groupwise
> has become sluggish. Over a period of 5 or so days it becomes less
> responsive. Now I cant even get top or df h to activate from the
> terminal or from an ssh session. The groupwise system is running
> fine, but the server seems busy. What can I do to investigate why it
> is not responsive to terminal commands? Yesterday I did a restart of
> the groupwise system and it took 5 or 6 minutes to restart and it
> only has 100 users, a small system. I appreciate your help.

The first thing we have to determine is whether this issue lies within
your Dom0 or your DomU. If only one DomU is affected, I would suspect
the DomU (Server 2?).

Please run these commands on your DomU:


fdisk -l
cat /etc/fstab



[CODE]
groupwise:~ # fdisk -l

Disk /dev/xvda: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders, total 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00007bc8

Device Boot Start End Blocks Id System
/dev/xvda1 2048 4208639 2103296 82 Linux swap / Solaris
/dev/xvda2 * 4208640 41943039 18867200 83 Linux

Disk /dev/xvdb: 4189 MB, 4189161472 bytes
255 heads, 63 sectors/track, 509 cylinders, total 8181956 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/xvdb doesn't contain a valid partition table

Disk /dev/xvdd: 96.8 GB, 96762161664 bytes
255 heads, 63 sectors/track, 11763 cylinders, total 188988597 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0001b841

Device Boot Start End Blocks Id System
/dev/xvdd1 2048 188987391 94492672 83 Linux
groupwise:~ # cat etc/fstab
cat: etc/fstab: No such file or directory
groupwise:~ # cat /etc/fstab
/dev/xvda1 swap swap defaults 0 0
/dev/xvda2 / ext3 acl,user_xattr 1 1
proc /proc proc defaults 0 0
sysfs /sys sysfs noauto 0 0
debugfs /sys/kernel/debug debugfs noauto 0 0
devpts /dev/pts devpts mode=0620,gid=5 0 0
/dev/xvdd1 /mnt/vol ext3 acl,user_xattr 1 2
198.nnn.nnn.209:/media/nss/SVUSD1DATA/stu-domain /media/nss/SVUSD1DATA/stu-domain nfs defaults 0 0
groupwise:~ #

[CODE]

If you shutdown GroupWise, is your server more responsive?

Did you move GWIA as you planned to do in your OP?

Poor performance can be caused by IO bottlenecks. Have you checked for
excessive swapping on each server or verified that there are no disk
errors that are causing excessive retries?

One other suggestion: If this is a production server and you need to
get this resolved quickly, have you considered opening a Service
Request to get some hands-on support?

Please provide the requested information to get a better understanding
of what's happening.

I'll post more later today.

KBOYLE
27-Sep-2013, 08:02
I'm going to summarize what you told me in your four previous posts and hopefully consolidate all the important facts in one place. We still need some additional info which I hope you can provide. This way, anyone who has an idea or suggestion can jump in.


I have two oes11 sp1 servers virtualized on a xen sles 11 sp 2 server

Xen host (SLES-XEN):

First, the physical server has a RAID 1 hardware type and a hardware type RAID 5.


SLES-XEN:/ # fdisk -l

Disk /dev/cciss/c0d0: 146.8 GB, 146778685440 bytes
255 heads, 63 sectors/track, 17844 cylinders, total 286677120 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00075c47

Device Boot Start End Blocks Id System
/dev/cciss/c0d0p1 63 4209029 2104483+ 82 Linux swap / Solaris
/dev/cciss/c0d0p2 * 4209030 286663859 141227415 83 Linux

Disk /dev/cciss/c0d1: 899.9 GB, 899898718208 bytes
255 heads, 63 sectors/track, 109406 cylinders, total 1757614684 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0006a452

Device Boot Start End Blocks Id System
/dev/cciss/c0d1p1 32 706876064 353438016+ 5 Extended
/dev/cciss/c0d1p2 706876065 1757607389 525365662+ 83 Linux
/dev/cciss/c0d1p5 64 195309599 97654768 83 Linux
/dev/cciss/c0d1p6 195309632 292968479 48829424 83 Linux
/dev/cciss/c0d1p7 292977468 481966064 94494298+ 83 Linux



SLES-XEN:/ # cat /etc/fstab
/dev/disk/by-id/cciss-3600508b1001032373120202020200004-part2 / reiserfs acl,user_xattr 1 1
/dev/disk/by-id/cciss-3600508b1001032373120202020200004-part1 swap swap defaults 0 0
proc /proc proc defaults 0 0
sysfs /sys sysfs noauto 0 0
debugfs /sys/kernel/debug debugfs noauto 0 0
usbfs /proc/bus/usb usbfs noauto 0 0
devpts /dev/pts devpts mode=0620,gid=5 0 0
/dev/disk/by-id/cciss-3600508b1001032373120202020200005-part4 /mnt/mount ext3 rw 0 0
##/dev/cciss/c0d1p4 /mnt/mount ext3 rw 0 0
##next line edited by jmcg 21May2012 so partition not mounted
##/dev/disk/by-id/cciss-3600508b1001032373120202020200005-part3 /mnt/mount2 ext3 acl,user_xattr 1 2
/dev/disk/by-id/cciss-3600508b1001032373120202020200005-part6 /mnt/mount3 ext3 acl,user_xattr 1 2
SLES-XEN:/ #


Each DomU (guests or vmís) are installed as disk files on the RAID 1. That is, not only is the Host OS installed on the RAID 1 but also the disk files for each of the DomUís (they are in the /var/lib/xen/images/ directory) are too, but they are technically files (disk images).
Okay, your RAID 1 array has a swap partition and one partition for /. All the Xen host system is contained in that one partition, including /var/lib/xen/images/. If you are seeing disk utilisation of 78% (or 86%), that is an issue that needs to be looked at.


The RAID 5 shows up in the host directory (/dev/cciss/c0d1x) with each of the partitions (linux) that I have created for the DomUís to use. Each DomU has a partition of the RAID 5 attached to it. They show up as /dev/cciss/c0d1p2 or 3 or 4, depending on the order I created them.

Within each of the domUís, I see those partitions (RAID 5) as xvdb or xvdd or whatever. That is what I see from the terminal of the domU when I enter df Ėh. I donít see them from the Dom0ís terminal because they are not attached to it, only to the DomUís.

According to your fstab I see:

/dev/disk/by-id/cciss-3600508b1001032373120202020200005-part4 /mnt/mount
/dev/disk/by-id/cciss-3600508b1001032373120202020200005-part6 /mnt/mount3



That clearly contradicts what you say. The device at /mnt/mount does not appear to exist but I'm more concerned with the device at /mnt/mount3. What is that device used for? Is it also assigned to a DomU?

DomU server 2 (groupwise):

groupwise:~ # fdisk -l

Disk /dev/xvda: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders, total 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00007bc8

Device Boot Start End Blocks Id System
/dev/xvda1 2048 4208639 2103296 82 Linux swap / Solaris
/dev/xvda2 * 4208640 41943039 18867200 83 Linux

Disk /dev/xvdb: 4189 MB, 4189161472 bytes
255 heads, 63 sectors/track, 509 cylinders, total 8181956 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/xvdb doesn't contain a valid partition table

Disk /dev/xvdd: 96.8 GB, 96762161664 bytes
255 heads, 63 sectors/track, 11763 cylinders, total 188988597 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0001b841

Device Boot Start End Blocks Id System
/dev/xvdd1 2048 188987391 94492672 83 Linux


groupwise:~ # cat /etc/fstab
/dev/xvda1 swap swap defaults 0 0
/dev/xvda2 / ext3 acl,user_xattr 1 1
proc /proc proc defaults 0 0
sysfs /sys sysfs noauto 0 0
debugfs /sys/kernel/debug debugfs noauto 0 0
devpts /dev/pts devpts mode=0620,gid=5 0 0
/dev/xvdd1 /mnt/vol ext3 acl,user_xattr 1 2
198.nnn.nnn.209:/media/nss/SVUSD1DATA/stu-domain /media/nss/SVUSD1DATA/stu-domain nfs defaults 0 0
groupwise:~ #


Server 2
Examine the /dev/disk/by-path I see the xvdb which is not shown in the output from df -h. I also see the xvdd in both the /dev/disk/by-path as is expected. Yet the xvdb is a path that can be drilled down into from the terminal or the gui.

The bad part is this xvdb has the directory for groupwise gwia. My plan tonight is to stop gw and to copy it from the xvdb seen by server 2 to the xvdd on server 2. Change the path in C1 for gwia, start it back up, that should take care of that, but....

If you look at your fdisk output you'll see that "Disk /dev/xvdb doesn't contain a valid partition table" and according to fstab, it isn't mounted. Of course you would need a valid filesystem to mount it.

What filesystem do you expect it should contain?
How did you create it?
How are you able to drill down into it as you state above?



Please explain what you are doing with /media/nss/SVUSD1DATA/stu-domain. It looks as if you are mounting a remote NSS volume yet you say:

Kevin,
I won't bore you with my details, but the bottom line is that the two servers do not see the same disk.

I know you have described other symptoms and have performance issues but before we get to them we need answers to these questions.

jmozdzen
29-Sep-2013, 23:17
Hi Kevin, hi jmcg,


I'm going to summarize what you told me in your four previous posts and hopefully consolidate all the important facts in one place. We still need some additional info which I hope you can provide. This way, anyone who has an idea or suggestion can jump in.
[...]

I've been following this thread and was/am confused by the diverging infos, too. May I ask you (jmcg) to provide a current output of "mount" from the three systems as well? Noticing the comment characters in fstab and the entry for a non-existing partition makes it clear that the actual mount situation may differ from the way it is set up in the files.

Additionally, I'd like to ask for at least the disk definitions for the two DomUs, as contained in the DomUs' config files (or persisted configurations), plus "ls -l /var/lib/xen/images" from the Dom0 (where the disk images files are said to be stored).

This all just to clear up the potentially wrong conclusions that can be drawn from the information provided so far.

Regards,
Jens

jmcg
04-Oct-2013, 21:01
I'm not confident the code box will be right. Hopefully you can see what you have asked for. If you need more information, please let me know.

From the host




SLES-XEN:/var/lib/xen/images # mount -l
/dev/cciss/c0d0p2 on / type reiserfs (rw,acl,user_xattr)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
debugfs on /sys/kernel/debug type debugfs (rw)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,mode=1777)
devpts on /dev/pts type devpts (rw,mode=0620,gid=5)
fusectl on /sys/fs/fuse/connections type fusectl (rw)
none on /var/lib/ntp/proc type proc (ro,nosuid,nodev)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
gvfs-fuse-daemon on /root/.gvfs type fuse.gvfs-fuse-daemon (rw,nosuid,nodev)
/dev/sr0 on /media/O type udf (ro,nosuid,nodev,uid=0) [O]
/root/Desktop/OES11-SP1-addon_with_SLES11-SP2-x86_64-DVD.iso on /nfs type iso9660 (ro) [CDROM]
nfsd on /proc/fs/nfsd type nfsd (rw)
198.nnn.nnn.250:/nfs on /tmp/tmpcOUDYL type nfs (rw,addr=198.nnn.nnn.250)
SLES-XEN:/var/lib/xen/images # ls -l /var/lib/xen/images/
total 4
-rw-r--r-- 1 root root 65 Sep 20 08:57 .disk-list
drwxr-xr-x 2 root root 72 May 21 2012 WinSrvr1
drwxr-xr-x 2 root root 80 Feb 19 2013 grpwisenew
drwxr-xr-x 2 root root 152 Sep 20 09:15 stu-grpwise
drwxr-xr-x 2 root root 80 May 16 2012 student-ehs-server
SLES-XEN:/var/lib/xen/images # cd ../

SLES-XEN:/etc/xen/vm # ls
grpwisenew sles10-rescue stu-grpwise.xml
grpwisenew.xml sles10-rescue1 student-ehs-server
old stu-grpwise student-ehs-server.xml
SLES-XEN:/etc/xen/vm # cat grpwisenew
name="grpwisenew"
description="New grp wise server to install gw 2012 on "
uuid="3d7fb292-e28d-ee75-6a0b-854b0ff1ee6c"
memory=4000
maxmem=4000
vcpus=1
on_poweroff="destroy"
on_reboot="restart"
on_crash="destroy"
localtime=0
keymap="en-us"
builder="linux"
bootloader="/usr/bin/pygrub"
bootargs=""
extra=" "
disk=[ 'file:/var/lib/xen/images/grpwisenew/disk0.raw,xvda,w', 'file:/root/Desktop/OES11-SP1-addon_with_SLES11-SP2-x86_64-DVD.iso,xvdb:cdrom,r', ]
vif=[ 'mac=00:16:3e:74:77:91,bridge=br0', ]
vfb=['type=vnc,vncunused=1']
SLES-XEN:/etc/xen/vm # cat stu-grpwise
name="stu-grpwise"
description="Server for Studetn Groupwise domain and PO"
uuid="4ba7ac47-8cb6-ab99-c103-e50eefbf4496"
memory=4000
maxmem=4000
vcpus=1
on_poweroff="destroy"
on_reboot="restart"
on_crash="destroy"
localtime=0
keymap="en-us"
builder="linux"
bootloader="/usr/bin/pygrub"
bootargs=""
extra=" hostip=198.nnn.nnn.209 netmask=255.255.255.0 gateway=198.nnn.nnn.1"
disk=[ 'file:/var/lib/xen/images/stu-grpwise/disk0.raw,xvda,w', ]
vif=[ 'mac=00:16:3e:5e:37:09,bridge=br0', ]
vfb=['type=vnc,vncunused=1']
SLES-XEN:/etc/xen/vm # fdisk --help
fdisk: invalid option -- '-'
Usage:
fdisk [options] <disk> change partition table
fdisk [options] -l <disk> list partition table(s)
fdisk -s <partition> give partition size(s) in blocks

Options:
-b <size> sector size (512, 1024, 2048 or 4096)
-c[=<mode>] compatible mode: 'dos' or 'nondos' (default)
-h print this help text
-u[=<unit>] display units: 'cylinders' or 'sectors' (default)
-v print program version
-C <number> specify the number of cylinders
-H <number> specify the number of heads
-S <number> specify the number of sectors per track

SLES-XEN:/etc/xen/vm # fdisk -l

Disk /dev/cciss/c0d0: 146.8 GB, 146778685440 bytes
255 heads, 63 sectors/track, 17844 cylinders, total 286677120 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00075c47

Device Boot Start End Blocks Id System
/dev/cciss/c0d0p1 63 4209029 2104483+ 82 Linux swap / Solaris
/dev/cciss/c0d0p2 * 4209030 286663859 141227415 83 Linux

Disk /dev/cciss/c0d1: 899.9 GB, 899898718208 bytes
255 heads, 63 sectors/track, 109406 cylinders, total 1757614684 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0006a452

Device Boot Start End Blocks Id System
/dev/cciss/c0d1p1 32 706876064 353438016+ 5 Extended
/dev/cciss/c0d1p2 706876065 1757607389 525365662+ 83 Linux
/dev/cciss/c0d1p5 64 195309599 97654768 83 Linux
/dev/cciss/c0d1p6 195309632 292968479 48829424 83 Linux
/dev/cciss/c0d1p7 292977468 481966064 94494298+ 83 Linux
SLES-XEN:/etc/xen/vm #


From DomU=grpwisenew


groupwise:~/Desktop # mount -l
/dev/xvda2 on / type ext3 (rw,acl,user_xattr)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
debugfs on /sys/kernel/debug type debugfs (rw)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,mode=1777)
devpts on /dev/pts type devpts (rw,mode=0620,gid=5)
/dev/xvdd1 on /mnt/vol type ext3 (rw,acl,user_xattr)
fusectl on /sys/fs/fuse/connections type fusectl (rw)
securityfs on /sys/kernel/security type securityfs (rw)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
198.nnn.nnn.209:/media/nss/SVUSD1DATA/stu-domain on /media/nss/SVUSD1DATA/stu-domain type nfs (rw,addr=198.nnn.nnn.209)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
none on /var/lib/ntp/proc type proc (ro,nosuid,nodev)
nfsd on /proc/fs/nfsd type nfsd (rw)
admin on /_admin type nssadmin (rw)
novfs on /var/opt/novell/nclmnt type novfs (rw)
gvfs-fuse-daemon on /root/.gvfs type fuse.gvfs-fuse-daemon (rw,nosuid,nodev)
groupwise:~/Desktop # top

groupwise:~/Desktop # fdisk -l

Disk /dev/xvda: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders, total 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00007bc8

Device Boot Start End Blocks Id System
/dev/xvda1 2048 4208639 2103296 82 Linux swap / Solaris
/dev/xvda2 * 4208640 41943039 18867200 83 Linux

Disk /dev/xvdb: 4189 MB, 4189161472 bytes
255 heads, 63 sectors/track, 509 cylinders, total 8181956 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/xvdb doesn't contain a valid partition table

Disk /dev/xvdd: 96.8 GB, 96762161664 bytes
255 heads, 63 sectors/track, 11763 cylinders, total 188988597 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0001b841

Device Boot Start End Blocks Id System
/dev/xvdd1 2048 188987391 94492672 83 Linux
groupwise:~/Desktop #



From Domu=stu-grpwise


login as: root
Using keyboard-interactive authentication.
Password:
Last login: Tue Sep 24 07:53:45 2013 from ehs11.sisnet.ssku.k12.ca.us
stu-grpwise:~ # mount -l
/dev/xvda2 on / type ext3 (rw,acl,user_xattr)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
debugfs on /sys/kernel/debug type debugfs (rw)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,mode=1777)
devpts on /dev/pts type devpts (rw,mode=0620,gid=5)
fusectl on /sys/fs/fuse/connections type fusectl (rw)
securityfs on /sys/kernel/security type securityfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
none on /var/lib/ntp/proc type proc (ro,nosuid,nodev)
admin on /_admin type nssadmin (rw)
/dev/pool/SVUSD1DATA on /opt/novell/nss/mnt/.pools/SVUSD1DATA type nsspool (rw,name=SVUSD1DATA)
SVUSD1DATA on /media/nss/SVUSD1DATA type nssvol (rw,name=SVUSD1DATA)
novfs on /var/opt/novell/nclmnt type novfs (rw)
gvfs-fuse-daemon on /root/.gvfs type fuse.gvfs-fuse-daemon (rw,nosuid,nodev)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)
mail.svusd.us:/mnt/vol/gwdomain on /mnt/vol/gwdomain type nfs (rw,addr=198.nnn.nnn.240)
stu-grpwise:~ # fdisk -l

Disk /dev/xvda: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders, total 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000dfc59

Device Boot Start End Blocks Id System
/dev/xvda1 2048 4208639 2103296 82 Linux swap / Solaris
/dev/xvda2 * 4208640 41943039 18867200 83 Linux

Disk /dev/xvdb: 100.0 GB, 99998482432 bytes
255 heads, 32 sectors/track, 23934 cylinders, total 195309536 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Device Boot Start End Blocks Id System
/dev/xvdb1 32 195301439 97650704 65 Novell Netware 386

Disk /dev/mapper/SVUSD1DATA: 100.0 GB, 99994304512 bytes
255 heads, 63 sectors/track, 12156 cylinders, total 195301376 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x42530000

Disk /dev/mapper/SVUSD1DATA doesn't contain a valid partition table
stu-grpwise:~ #

KBOYLE
11-Oct-2013, 21:29
There is a lot of data here. To make it easier to discuss, I will create a separate post for each server. If Jens or anyone else has any ideas, please jump in.

Xen host (SLES-XEN):

Items I have highlighted in green look fine. Items I have highlighted in red require some additional explanation.


SLES-XEN:/ # cat /etc/fstab
/dev/disk/by-id/cciss-3600508b1001032373120202020200004-part2 / reiserfs acl,user_xattr 1 1
/dev/disk/by-id/cciss-3600508b1001032373120202020200004-part1 swap swap defaults 0 0
proc /proc proc defaults 0 0
sysfs /sys sysfs noauto 0 0
debugfs /sys/kernel/debug debugfs noauto 0 0
usbfs /proc/bus/usb usbfs noauto 0 0
devpts /dev/pts devpts mode=0620,gid=5 0 0
/dev/disk/by-id/cciss-3600508b1001032373120202020200005-part4 /mnt/mount ext3 rw 0 0
##/dev/cciss/c0d1p4 /mnt/mount ext3 rw 0 0
##next line edited by jmcg 21May2012 so partition not mounted
##/dev/disk/by-id/cciss-3600508b1001032373120202020200005-part3 /mnt/mount2 ext3 acl,user_xattr 1 2
/dev/disk/by-id/cciss-3600508b1001032373120202020200005-part6 /mnt/mount3 ext3 acl,user_xattr 1 2
SLES-XEN:/ #

These two items appear in /etc/fstab but are not mounted. What do they represent? If they are not needed, they should be removed from fstab.

/dev/disk/by-id/cciss-3600508b1001032373120202020200005-part4 /mnt/mount
/dev/disk/by-id/cciss-3600508b1001032373120202020200005-part6 /mnt/mount3




SLES-XEN:/var/lib/xen/images # mount -l
/dev/cciss/c0d0p2 on / type reiserfs (rw,acl,user_xattr)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
debugfs on /sys/kernel/debug type debugfs (rw)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,mode=1777)
devpts on /dev/pts type devpts (rw,mode=0620,gid=5)
fusectl on /sys/fs/fuse/connections type fusectl (rw)
none on /var/lib/ntp/proc type proc (ro,nosuid,nodev)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
gvfs-fuse-daemon on /root/.gvfs type fuse.gvfs-fuse-daemon (rw,nosuid,nodev)
/dev/sr0 on /media/O type udf (ro,nosuid,nodev,uid=0) [O]
/root/Desktop/OES11-SP1-addon_with_SLES11-SP2-x86_64-DVD.iso on /nfs type iso9660 (ro) [CDROM]
nfsd on /proc/fs/nfsd type nfsd (rw)
198.nnn.nnn.250:/nfs on /tmp/tmpcOUDYL type nfs (rw,addr=198.nnn.nnn.250)

These devices do not appear in /etc/fstab:

OES11-SP1-addon_with_SLES11-SP2-x86_64-DVD.iso on your root Desktop which you have mounted on /nfs. Please explain why you have done this and what you hope to accomplish.

198.nnn.nnn.250:/nfs is mounted on /tmp/tmpcOUDYL. Please explain why you have done this and what you hope to accomplish.

KBOYLE
11-Oct-2013, 22:11
DomU Analysis


SLES-XEN:/var/lib/xen/images # ls -l /var/lib/xen/images/
total 4
-rw-r--r-- 1 root root 65 Sep 20 08:57 .disk-list
drwxr-xr-x 2 root root 72 May 21 2012 WinSrvr1
drwxr-xr-x 2 root root 80 Feb 19 2013 grpwisenew
drwxr-xr-x 2 root root 152 Sep 20 09:15 stu-grpwise
drwxr-xr-x 2 root root 80 May 16 2012 student-ehs-server


According to this directory listing, you may have image files for four DomUs. If WinSrvr1 no longer exists, if that directory still contains an disk image file, and if you no longer need it, you can free up some space by deleting it.

Note:
When you delete a DomU (xm delete <DomU>) it removes the DomU's definition from the XenStore but it does not delete the disk image file.


SLES-XEN:/etc/xen/vm # ls
grpwisenew sles10-rescue stu-grpwise.xml
grpwisenew.xml sles10-rescue1 student-ehs-server
old stu-grpwise student-ehs-server.xml
SLES-XEN:/etc/xen/vm # cat grpwisenew

This shows definitions for five DomUs:

grpwisenew
stu-grpwise
student-ehs-server
sles10-rescue
sles10-rescue1
old


Note:
These files are created when you create a new DomU. The definitions are then imported into the XenStore. After that, these files are not maintained or updated. For example, any changes you make to a DomU using the Virtual Machine Manager are not reflected in these files. To view the current definition for a DomU, use "xm list -l <DomU>"

If you want a better understanding how this works, see my article:
Managing your Xen DomU: Difference between "xm new" and "xm create" (https://forums.suse.com/content.php?125-Managing-your-Xen-DomU-Difference-between-xm-new-and-xm-create)

KBOYLE
11-Oct-2013, 23:54
DomU server 2 (groupwise):

I'm going to add your "mount -l" to the information from my earlier post. Please provide the information I requested.

groupwise:~ # fdisk -l

Disk /dev/xvda: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders, total 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00007bc8

Device Boot Start End Blocks Id System
/dev/xvda1 2048 4208639 2103296 82 Linux swap / Solaris
/dev/xvda2 * 4208640 41943039 18867200 83 Linux

Disk /dev/xvdb: 4189 MB, 4189161472 bytes
255 heads, 63 sectors/track, 509 cylinders, total 8181956 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/xvdb doesn't contain a valid partition table

Disk /dev/xvdd: 96.8 GB, 96762161664 bytes
255 heads, 63 sectors/track, 11763 cylinders, total 188988597 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0001b841

Device Boot Start End Blocks Id System
/dev/xvdd1 2048 188987391 94492672 83 Linux



groupwise:~ # cat /etc/fstab
/dev/xvda1 swap swap defaults 0 0
/dev/xvda2 / ext3 acl,user_xattr 1 1
proc /proc proc defaults 0 0
sysfs /sys sysfs noauto 0 0
debugfs /sys/kernel/debug debugfs noauto 0 0
devpts /dev/pts devpts mode=0620,gid=5 0 0
/dev/xvdd1 /mnt/vol ext3 acl,user_xattr 1 2
198.nnn.nnn.209:/media/nss/SVUSD1DATA/stu-domain /media/nss/SVUSD1DATA/stu-domain nfs defaults 0 0
groupwise:~ #


groupwise:~/Desktop # mount -l
/dev/xvda2 on / type ext3 (rw,acl,user_xattr)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
debugfs on /sys/kernel/debug type debugfs (rw)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,mode=1777)
devpts on /dev/pts type devpts (rw,mode=0620,gid=5)
/dev/xvdd1 on /mnt/vol type ext3 (rw,acl,user_xattr)
fusectl on /sys/fs/fuse/connections type fusectl (rw)
securityfs on /sys/kernel/security type securityfs (rw)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
198.nnn.nnn.209:/media/nss/SVUSD1DATA/stu-domain on /media/nss/SVUSD1DATA/stu-domain type nfs (rw,addr=198.nnn.nnn.209)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
none on /var/lib/ntp/proc type proc (ro,nosuid,nodev)
nfsd on /proc/fs/nfsd type nfsd (rw)
admin on /_admin type nssadmin (rw)
novfs on /var/opt/novell/nclmnt type novfs (rw)
gvfs-fuse-daemon on /root/.gvfs type fuse.gvfs-fuse-daemon (rw,nosuid,nodev)
groupwise:~/Desktop #


Server 2
Examine the /dev/disk/by-path I see the xvdb which is not shown in the output from df -h. I also see the xvdd in both the /dev/disk/by-path as is expected. Yet the xvdb is a path that can be drilled down into from the terminal or the gui.

The bad part is this xvdb has the directory for groupwise gwia. My plan tonight is to stop gw and to copy it from the xvdb seen by server 2 to the xvdd on server 2. Change the path in C1 for gwia, start it back up, that should take care of that, but....

If you look at your fdisk output you'll see that "Disk /dev/xvdb doesn't contain a valid partition table" and according to fstab, it isn't mounted. Of course you would need a valid filesystem to mount it.


What filesystem do you expect it should contain?
How did you create it?
How are you able to drill down into it as you state above?



Kevin,
I won't bore you with my details, but the bottom line is that the two servers do not see the same disk.
Please explain what you are doing with /media/nss/SVUSD1DATA/stu-domain. The third code box clearly shows that it is mounted on /media/nss/SVUSD1DATA/stu-domain.

We need to know what you are trying to do before we can provide any suggestions on how to improve things. If you have already resolved your issues, please let us know.

KBOYLE
12-Oct-2013, 17:22
DomU Server 1 (stu-grpwise):

stu-grpwise:~ # fdisk -l

Disk /dev/xvda: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders, total 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000dfc59

Device Boot Start End Blocks Id System
/dev/xvda1 2048 4208639 2103296 82 Linux swap / Solaris
/dev/xvda2 * 4208640 41943039 18867200 83 Linux

Disk /dev/xvdb: 100.0 GB, 99998482432 bytes
255 heads, 32 sectors/track, 23934 cylinders, total 195309536 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Device Boot Start End Blocks Id System
/dev/xvdb1 32 195301439 97650704 65 Novell Netware 386

Disk /dev/mapper/SVUSD1DATA: 100.0 GB, 99994304512 bytes
255 heads, 63 sectors/track, 12156 cylinders, total 195301376 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x42530000

Disk /dev/mapper/SVUSD1DATA doesn't contain a valid partition table
stu-grpwise:~ #


stu-grpwise:~ # mount -l
/dev/xvda2 on / type ext3 (rw,acl,user_xattr)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
debugfs on /sys/kernel/debug type debugfs (rw)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,mode=1777)
devpts on /dev/pts type devpts (rw,mode=0620,gid=5)
fusectl on /sys/fs/fuse/connections type fusectl (rw)
securityfs on /sys/kernel/security type securityfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
none on /var/lib/ntp/proc type proc (ro,nosuid,nodev)
admin on /_admin type nssadmin (rw)
/dev/pool/SVUSD1DATA on /opt/novell/nss/mnt/.pools/SVUSD1DATA type nsspool (rw,name=SVUSD1DATA)
SVUSD1DATA on /media/nss/SVUSD1DATA type nssvol (rw,name=SVUSD1DATA)
novfs on /var/opt/novell/nclmnt type novfs (rw)
gvfs-fuse-daemon on /root/.gvfs type fuse.gvfs-fuse-daemon (rw,nosuid,nodev)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)
mail.svusd.us:/mnt/vol/gwdomain on /mnt/vol/gwdomain type nfs (rw,addr=198.nnn.nnn.240)
stu-grpwise:~ #



Server 1
Examine the /dev/disk/by-path I see the xvdb which is the nss volume. That is fine. When I examine the output from df -h command it shows up as expected.

That confirms what I see too:

/dev/xvdb is 100 GB.
/dev/xvdb1 shows partition type is Novell Netware 386.
/dev/pool/SVUSD1DATA is your NSS Pool. Although the info you provided doesn't show it, I assume as you stated it is using /dev/xvdb1.
SVUSD1DATA is your only NSS volume mounted on /media/nss/SVUSD1DATA.

All of this looks fine although I normally use different names for my pools and volumes just to avoid confusion. Disk /dev/mapper/SVUSD1DATA doesn't contain a valid partition table because it is used by NSS.

The last item requires some additional explanation: mail.svusd.us:/mnt/vol/gwdomain. I assume from its name it contains your GroupWise domain.

What is the source server?
What is running on that server?
If it is one of the servers we are discussing here, which device actually contains the data.?

KBOYLE
12-Oct-2013, 18:19
I have just posted my comments about your DomU Server 1 (stu-grpwise) and have gained some additional insight.

It would seem that the mounted device "198.nnn.nnn.209:/media/nss/SVUSD1DATA/stu-domain" as shown in your fstab and your list of mounted devices is actually a directory on your NSS volume "/media/nss/SVUSD1DATA" which is mounted and, presumably, being used on your "stu-grpwise" server. If that is true, then it is an issue that requires your attention.

Disk /dev/xvdb doesn't contain a valid partition table. We see the same message on your stu-grpwise DomU but concluded it was okay because it was used by NSS. On this server I see no NSS pools or NSS volumes in your fstab or among your mounted devices which raises some additional questions:

Where did /dev/xvdb come from?
Was it added to the DomU as an additional device after the DomU was created?
Does it, in fact, contain an NSS filesystem? If so, how/where was it created?
Since you haven't provided current configurations for your DomUs, is it possible that the /dev/xvdb source device is the same source device used by /dev/xvdb in your stu-grpwise DomU?