Hi,
Quote Originally Posted by carnold6 View Post
Seems to be a whole lotta swapping going on?

Code:
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0  69568  36024     12 5275148    0    0   605   247   19   15  2  1 89  8  0
 0  0  69568  33248     12 5277308    0    0  2048  3756 9727 6364  3  2 94  1  1
 0  0  69568  33776     12 5276992    0    0  4096     0 12421 8200  2  1 96  0  0
 0  0  69568  33248     12 5277128    0    0     0     0 12375 7601  1  2 96  0  0
 0  0  69568  35264     12 5275176    0    0  2048   496 8858 5668  2  1 96  0  0
 0  0  69568  33520     12 5277108    0    0  2048     0 7815 4995  2  1 97  0  0
 1  0  69568  33680     12 5277120    0    0     0   564 12163 7613  2  1 96  0  0
 0  0  69568  35364     12 5275232    0    0  2048     0 8938 5779  2  1 96  0  0
 0  0  69568  35568     12 5275120    0    0    32     0 11165 7067  2  1 96  0  0
 1  0  69568  35304     12 5275312    0    0  4064     0 11428 7212  1  1 96  1  1
 0  0  69568  31176     12 5279108    0    0  4096     0 10352 6522  2  2 96  0  0
 2  0  69568  35048     12 5275196    0    0     0   860 11276 7460  3  1 96  0  0
 0  0  69568  33912     12 5277228    0    0  2048     0 11314 7665  1  1 98  0  0
 0  0  69568  33984     12 5277196    0    0   140  8136 10859 7342  2  2 92  5  0
 0  0  69568  33832     12 5277192    0    0     0     0 15453 11446  4  2 93  0  1
 0  0  69568  31888     12 5279252    0    0  1908     0 12559 7889  1  1 97  1  0
 0  0  69568  36364     12 5273840    0    0  1880   544 11170 6972  1  1 94  4  0
 0  0  69568  32228     12 5277988    0    0  4256     0 12076 7457  1  1 98  0  0
 0  0  69568  30248     12 5280056    0    0  2048     0 10443 6423  0  1 98  0  0
 0  0  69568  30248     12 5280060    0    0    32     0 9347 5848  1  1 98  0  0
 0  1  69568  35212     12 5273668    0    0  4704     0 10905 7029  1  1 96  2  0
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  0  69560  35768     12 5273820    0    0    80   488 12882 8007  0  1 97  1  0
 0  0  69560  31752     12 5277860    0    0  4096     0 11485 6982  1  1 97  1  0
 1  0  69560  31800     12 5277900    0    0     0     0 14506 8909  1  1 97  0  0
 0  0  69560  38400     12 5271000    0    0  2188     8 10443 6672  2  1 96  0  0
 1  1  69560  34500     12 5272536    0    0  1788     0 9969 6969  6  2 90  2  0
 0  1  69560  33372     12 5266800    0    0 21168   160 12942 8493  7  2 77 13  0
 1  0  69560  38816     12 5261724    0    0  5632     0 8764 6308  9  1 83  6  0
 0  0  69560  34832     12 5265868    0    0  4064     0 13449 9003  4  2 94  0  0
 1  0  69560  34800     12 5265868    0    0    32   296 18274 12935  4  2 93  0  1
 0  0  69560  32384     12 5268032    0    0  2048   336 13609 9179  2  2 96  0  1
 0  0  69560  32480     12 5268084    0    0     0 15152 16132 10623  1  2 96  0  1
 1  0  69560  37416     12 5262936    0    0  4064    68 9962 6314  1  1 92  6  0
 0  0  69560  37224     12 5263056    0    0    84    68 10148 6466  1  1 97  1  0
 1  0  69560  33000     12 5267068    0    0  4012     0 12332 7714  1  1 97  0  1
 1  0  69560  36768     12 5263184    0    0   220     0 9300 6322  6  1 91  1  0
 2  0  69560  32784     12 5267288    0    0  4096   280 11401 7551  3  2 95  0  0
 0  0  69560  36616     12 5262044    0    0    84     0 11856 7624  5  2 92  0  0
 0  0  69560  32696     12 5266200    0    0  4012     0 10369 6651  1  1 98  0  0
 0  0  69560  32736     12 5266124    0    0     0     0 10167 6567  1  1 97  0  1
 1  0  69560  30944     12 5268164    0    0  2048     0 12373 7664  1  1 98  0  1
 0  0  69560  36024     12 5262964    0    0  4096   140 10867 6760  1  2 97  1  0
no, actually there's (during normal operations) *no* swapping going on, and that's good.

Here's how I read that output:

procs: let's ignore that for now

memory: while there's some "swapped" memory reported, this only tells that at some point in time the kernel decided to swap out some (at that time) unused stuff, in favor of making better use of that physical memory. There's still some "free" memory (only saying there's no operational pressure on physical memory management), a bit is used for buffers (that number appears awfully small to me, I have no explanation for that) and a lot is used for caching. As any free memory is used for i/o caching, that big "cache" number seems to imply that you may have more memory committed to that Dom0 than actually required - but it's a trade-off versus block i/o (coming to that further below).

swap: Those zeros tell you that there's no current swap activity - nothing swapped in (stuff that got swapped out because of memory constraints, but now is needed again for current operations) nor swapped out (as memory might be tight - which obviously isn't)

io: actual i/o to local block devices. You're doing some reads, and there are some writes, too... nothing to worry about. Depending on access patterns, file system settings and available cache memory (see above), these numbers *might* get high because every read/write needs to go the the actual block device, rather than being served by the cache. In your case I doubt that, and because of the low percentage of "waits" (see below, CPU "wa" number) you're not having an issue there, anyhow.

system: "in" are interrupts, i.e. caused by devices signaling available data. according to my personal experience, these numbers are a bit high - but depending on your setup, this may be normal, rather than an indicator of real problems. I'd look at the numbers in /proc/interrupts, to get a feeling where these are coming from. "cs" are so called "context switches", telling you the scheduler let a different process get it's share of CPU.

cpu: "us" is "user space", hence programs you're running. "sy" is "system space", stuff that's being handled by the kernel. Then you have "idle times", which often (and in your case) is high and thus indicating that those CPUs are not doing much. More than enough horse-power for the actual workload. "wa" is the percentage of CPU time spent waiting for i/o to complete - for DomUs on local disks, that's definitely a number to watch. But in your case, watching gets mostly boring, only once that number show a significant value.

So looking at these numbers gives an impression of a Dom0 with too much available memory (which might help the VMs instead, I'd immediately turn it down by 2 or 3 GB). The CPUs are mostly bored, there's a bit of reading and writing to disk going on, but nothing to actually worry about. The overall i/o wait is at 8 percent (shown by the first line of output), *that* looks a bit high, but may be explainable. Those high number of interrupts may be a pointer to something, too, but I'm not sure about that.

You gave the VMs 4 GB each - I suspect that that's not enough to reach optimum performance, as the VMs itself (the Windows OS) might be forced to swap out memory during boot. This then might be the cause for those 8 percent over-all i/o wait reported by vmstat - you might want to take a look at vmstat's output during start of the VMs and check the i/o wait percentage at that time... if it's going high, I'd try to throw more memory at these VMs (of course, doing a similar analysis *inside* the VMs would be a better starting point, but might prove to be difficult during the startup phase). Swapping of a DomU does slow it down significantly, even worse than if the Dom0 needs to swap.

Hope this helps a bit to clear up the picture

Regards,
J