PDA

View Full Version : Problems with Shutdown/Boot in a multipathed Environment



kaiwindheuser
21-May-2012, 10:14
Hello!

We use SLES 11 SP2
Linux <HOSTNAME> 3.0.26-0.7-default #1 SMP Tue Apr 17 10:27:57 UTC 2012 (3829766) x86_64 x86_64 x86_64 GNU/Linux

on an Intel Modular Server with Dual Storage Controller Setup.

Content of /etc/multipath.conf:
blacklist {
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^(hd|xvd)[a-z][[0-9]*]"
devnode "^cciss!c[0-9]d[0-9]*[p[0-9]*]"
}

devices {
device {
vendor "Intel"
product "Multi-Flex"
path_grouping_policy "group_by_prio"
getuid_callout "/lib/udev/scsi_id -g -u /dev/%n"
prio "alua"
path_checker tur
path_selector "round-robin 0"
hardware_handler "1 alua"
failback immediate
rr_weight uniform
rr_min_io 100
no_path_retry queue
features "1 queue_if_no_path"
}
}



Output of "multipath -ll"
222d60001557a1912 dm-1 Intel,Multi-Flex
size=550G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='round-robin 0' prio=130 status=active
| `- 0:0:0:1 sdb 8:16 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 0:0:1:1 sdd 8:48 active ready running
2228900015565f670 dm-0 Intel,Multi-Flex
size=40G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='round-robin 0' prio=130 status=active
| `- 0:0:0:0 sda 8:0 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 0:0:1:0 sdc 8:32 active ready running




The redundant Storage connection just works fine. We can remove one of the controllers and the Server switches
to the other path.
But when we shutdown the server, SuSE Linux is not able to do a clean unmount.
The following message is printed to the console:

"Not shutting down MD Raid - reboot/halt scripts do this." missing
Removing multipath targets: May 21 08:53:52 | 22289xxxxx_part2: map in use
May 21 08:53:52 | failed to remove multipath map 22289xxxxx



As a consequence it happened one time, that on bootup, fsck (File-System-Check) found
an orphaned inode and remove it.

How can we configure SLES 11 SP2 that it does a clean shutdown? Or is this normal behaviour?

Thanks for your reply in advance!

amo_vzug
22-May-2012, 06:24
Hi

It's possible that mdadmd should shutdown the md-raid-devices.

Try enabling it:


# Just check, if it's already enabled within the runlevel 3 and 5
$ chkconfig --list mdadmd
mdadmd 0:off 1:off 2:off 3:off 4:off 5:off 6:off

If all runlevels are set to "off", then activate runlevel 3 and 5 with:


$ chkconfig mdadmd on

Perhaps this helps.
Kind regards,
Tom

kaiwindheuser
22-May-2012, 08:06
Thanks for your reply.
I tried it.



<HOSTNAME>:~ # chkconfig --list mdadmd
mdadmd 0:off 1:off 2:off 3:on 4:off 5:on 6:off



Sadly, it didn't solve the problem.
(rebootet several times, the unmount was never clean during the shutdown process)

amo_vzug
22-May-2012, 11:28
Are you using software-raid (dm)?

The message your receive is from the init-script '/etc/init.d/boot.md'. And it seems, that this script is enabled -> that's the reason, why if you reboot or shutdown your system, this script (and also this warning) is executed / shown.
The only thing what this script does - if runlevel 0 (shutdown) or runlevel 6 (reboot) are called (stop-section) - is send this message to the console:


...
...
stop)
echo -n "Not shutting down MD RAID - reboot/halt scripts do this."
...
...


So, if your not using software-raid, then you can disable the 'boot.md'-script with


$ chkconfig boot.md off


And then you can also deactivate the other script (because it didn't solved your problem):


$ chkconfig mdadmd off


Hope, this helps.
Regards,
Tom

kaiwindheuser
22-May-2012, 13:28
Thanks for you reply again, Tom.

I did both two steps.

The result is the following at shutdown:


Removing multipath targets: | 22289xxxxx_part2: map in use
failed to remove multipath map 22289xxxxx


Note: The Line "Not shutting down MD Raid - reboot/halt scripts do this." missing is not being displayed anymore

We have three partitions on our server:

Volume1
22289xxxxx_part1 ===> Swap-File
22289xxxxx_part2 ===> Root

Volume2
222d6xxxxx_part1 ===> Files

Only the Root-Partition has the unmount problems.

Maybe it is an issue regarding wait/delay time. Meaning that the shutdown/unmount
process would need more time than it's given by the system to orderly shut down?

amo_vzug
22-May-2012, 20:17
You can try adding an "lsof /dev/mapper/xxx >/tmp/lsof 2>&1" (instead of 'xxx' put your root-partition) in the '/etc/init.d/boot.multipath'-script (before the line '$PROGRAM -F'). Then you can see in the file '/tmp/lsof', which files still are open or in use.
Another hint is to look to the order of the "K"-Scripts in /etc/init.d/boot.d.

What's your output of 'chkconfig --list -A | grep -i multi'? Are you using the root-partition within LVM? Or as a normal primary-partition?

Can you then also try to remark the line 'features "1 queue_if_no_path"' in your multipath.conf?

From "man multipath.conf":


...
The usage of queue_if_no_path option can lead to D state processes being hung and not killable in situations where all the paths to the LUN go offline. It is advisable to use the no_path_retry option instead.
...

kaiwindheuser
23-May-2012, 09:04
You can try adding an "lsof /dev/mapper/xxx >/tmp/lsof 2>&1" (instead of 'xxx' put your root-partition) in the '/etc/init.d/boot.multipath'-script (before the line '$PROGRAM -F'). Then you can see in the file '/tmp/lsof', which files still are open or in use.

The result is the following:


COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
init 1 root cwd DIR 253,4 4096 2 /
init 1 root rtd DIR 253,4 4096 2 /
init 1 root txt REG 253,4 40784 376901 /sbin/init
init 1 root mem REG 253,4 19149 1703950 /lib64/libdl-2.11.3.so
init 1 root mem REG 253,4 1754140 1703944 /lib64/libc-2.11.3.so
init 1 root mem REG 253,4 244800 1704002 /lib64/libsepol.so.1
init 1 root mem REG 253,4 118080 1704003 /lib64/libselinux.so.1
init 1 root mem REG 253,4 151051 1706614 /lib64/ld-2.11.3.so
kthreadd 2 root cwd DIR 253,4 4096 2 /
kthreadd 2 root rtd DIR 253,4 4096 2 /
ksoftirqd 3 root cwd DIR 253,4 4096 2 /
ksoftirqd 3 root rtd DIR 253,4 4096 2 /
kworker/0 4 root cwd DIR 253,4 4096 2 /
kworker/0 4 root rtd DIR 253,4 4096 2 /
migration 6 root cwd DIR 253,4 4096 2 /
migration 6 root rtd DIR 253,4 4096 2 /
watchdog/ 7 root cwd DIR 253,4 4096 2 /
watchdog/ 7 root rtd DIR 253,4 4096 2 /
migration 8 root cwd DIR 253,4 4096 2 /
migration 8 root rtd DIR 253,4 4096 2 /
ksoftirqd 10 root cwd DIR 253,4 4096 2 /
ksoftirqd 10 root rtd DIR 253,4 4096 2 /
watchdog/ 12 root cwd DIR 253,4 4096 2 /
watchdog/ 12 root rtd DIR 253,4 4096 2 /
migration 13 root cwd DIR 253,4 4096 2 /
migration 13 root rtd DIR 253,4 4096 2 /
kworker/2 14 root cwd DIR 253,4 4096 2 /
kworker/2 14 root rtd DIR 253,4 4096 2 /
ksoftirqd 15 root cwd DIR 253,4 4096 2 /
ksoftirqd 15 root rtd DIR 253,4 4096 2 /
watchdog/ 16 root cwd DIR 253,4 4096 2 /
watchdog/ 16 root rtd DIR 253,4 4096 2 /
migration 17 root cwd DIR 253,4 4096 2 /
migration 17 root rtd DIR 253,4 4096 2 /
kworker/3 18 root cwd DIR 253,4 4096 2 /
kworker/3 18 root rtd DIR 253,4 4096 2 /
ksoftirqd 19 root cwd DIR 253,4 4096 2 /
ksoftirqd 19 root rtd DIR 253,4 4096 2 /
watchdog/ 20 root cwd DIR 253,4 4096 2 /
watchdog/ 20 root rtd DIR 253,4 4096 2 /
migration 21 root cwd DIR 253,4 4096 2 /
migration 21 root rtd DIR 253,4 4096 2 /
kworker/4 22 root cwd DIR 253,4 4096 2 /
kworker/4 22 root rtd DIR 253,4 4096 2 /
ksoftirqd 23 root cwd DIR 253,4 4096 2 /
ksoftirqd 23 root rtd DIR 253,4 4096 2 /
watchdog/ 24 root cwd DIR 253,4 4096 2 /
watchdog/ 24 root rtd DIR 253,4 4096 2 /
migration 25 root cwd DIR 253,4 4096 2 /
migration 25 root rtd DIR 253,4 4096 2 /
kworker/5 26 root cwd DIR 253,4 4096 2 /
kworker/5 26 root rtd DIR 253,4 4096 2 /
ksoftirqd 27 root cwd DIR 253,4 4096 2 /
ksoftirqd 27 root rtd DIR 253,4 4096 2 /
watchdog/ 28 root cwd DIR 253,4 4096 2 /
watchdog/ 28 root rtd DIR 253,4 4096 2 /
migration 29 root cwd DIR 253,4 4096 2 /
migration 29 root rtd DIR 253,4 4096 2 /
kworker/6 30 root cwd DIR 253,4 4096 2 /
kworker/6 30 root rtd DIR 253,4 4096 2 /
ksoftirqd 31 root cwd DIR 253,4 4096 2 /
ksoftirqd 31 root rtd DIR 253,4 4096 2 /
watchdog/ 32 root cwd DIR 253,4 4096 2 /
watchdog/ 32 root rtd DIR 253,4 4096 2 /
cpuset 37 root cwd DIR 253,4 4096 2 /
cpuset 37 root rtd DIR 253,4 4096 2 /
khelper 38 root cwd DIR 253,4 4096 2 /
khelper 38 root rtd DIR 253,4 4096 2 /
netns 39 root cwd DIR 253,4 4096 2 /
netns 39 root rtd DIR 253,4 4096 2 /
sync_supe 40 root cwd DIR 253,4 4096 2 /
sync_supe 40 root rtd DIR 253,4 4096 2 /
bdi-defau 41 root cwd DIR 253,4 4096 2 /
bdi-defau 41 root rtd DIR 253,4 4096 2 /
kintegrit 42 root cwd DIR 253,4 4096 2 /
kintegrit 42 root rtd DIR 253,4 4096 2 /
kblockd 43 root cwd DIR 253,4 4096 2 /
kblockd 43 root rtd DIR 253,4 4096 2 /
md 44 root cwd DIR 253,4 4096 2 /
md 44 root rtd DIR 253,4 4096 2 /
khungtask 45 root cwd DIR 253,4 4096 2 /
khungtask 45 root rtd DIR 253,4 4096 2 /
kworker/1 46 root cwd DIR 253,4 4096 2 /
kworker/1 46 root rtd DIR 253,4 4096 2 /
kswapd0 47 root cwd DIR 253,4 4096 2 /
kswapd0 47 root rtd DIR 253,4 4096 2 /
ksmd 48 root cwd DIR 253,4 4096 2 /
ksmd 48 root rtd DIR 253,4 4096 2 /
khugepage 49 root cwd DIR 253,4 4096 2 /
khugepage 49 root rtd DIR 253,4 4096 2 /
fsnotify_ 50 root cwd DIR 253,4 4096 2 /
fsnotify_ 50 root rtd DIR 253,4 4096 2 /
crypto 51 root cwd DIR 253,4 4096 2 /
crypto 51 root rtd DIR 253,4 4096 2 /
kthrotld 53 root cwd DIR 253,4 4096 2 /
kthrotld 53 root rtd DIR 253,4 4096 2 /
kpsmoused 54 root cwd DIR 253,4 4096 2 /
kpsmoused 54 root rtd DIR 253,4 4096 2 /
kworker/5 55 root cwd DIR 253,4 4096 2 /
kworker/5 55 root rtd DIR 253,4 4096 2 /
mpt_poll_ 100 root cwd DIR 253,4 4096 2 /
mpt_poll_ 100 root rtd DIR 253,4 4096 2 /
mpt/0 101 root cwd DIR 253,4 4096 2 /
mpt/0 101 root rtd DIR 253,4 4096 2 /
kworker/2 102 root cwd DIR 253,4 4096 2 /
kworker/2 102 root rtd DIR 253,4 4096 2 /
kworker/3 103 root cwd DIR 253,4 4096 2 /
kworker/3 103 root rtd DIR 253,4 4096 2 /
kworker/4 104 root cwd DIR 253,4 4096 2 /
kworker/4 104 root rtd DIR 253,4 4096 2 /
kworker/6 105 root cwd DIR 253,4 4096 2 /
kworker/6 105 root rtd DIR 253,4 4096 2 /
kworker/7 106 root cwd DIR 253,4 4096 2 /
kworker/7 106 root rtd DIR 253,4 4096 2 /
scsi_eh_0 107 root cwd DIR 253,4 4096 2 /
scsi_eh_0 107 root rtd DIR 253,4 4096 2 /
kworker/u 108 root cwd DIR 253,4 4096 2 /
kworker/u 108 root rtd DIR 253,4 4096 2 /
kmpathd 116 root cwd DIR 253,4 4096 2 /
kmpathd 116 root rtd DIR 253,4 4096 2 /
kmpath_ha 117 root cwd DIR 253,4 4096 2 /
kmpath_ha 117 root rtd DIR 253,4 4096 2 /
kmpath_rd 150 root cwd DIR 253,4 4096 2 /
kmpath_rd 150 root rtd DIR 253,4 4096 2 /
kworker/u 291 root cwd DIR 253,4 4096 2 /
kworker/u 291 root rtd DIR 253,4 4096 2 /
khubd 367 root cwd DIR 253,4 4096 2 /
khubd 367 root rtd DIR 253,4 4096 2 /
kdmflush 490 root cwd DIR 253,4 4096 2 /
kdmflush 490 root rtd DIR 253,4 4096 2 /
kdmflush 493 root cwd DIR 253,4 4096 2 /
kdmflush 493 root rtd DIR 253,4 4096 2 /
kdmflush 506 root cwd DIR 253,4 4096 2 /
kdmflush 506 root rtd DIR 253,4 4096 2 /
kdmflush 507 root cwd DIR 253,4 4096 2 /
kdmflush 507 root rtd DIR 253,4 4096 2 /
kdmflush 510 root cwd DIR 253,4 4096 2 /
kdmflush 510 root rtd DIR 253,4 4096 2 /
kjournald 659 root cwd DIR 253,4 4096 2 /
kjournald 659 root rtd DIR 253,4 4096 2 /
flush-253 672 root cwd DIR 253,4 4096 2 /
flush-253 672 root rtd DIR 253,4 4096 2 /
udevd 743 root cwd DIR 253,4 4096 2 /
udevd 743 root rtd DIR 253,4 4096 2 /
udevd 743 root txt REG 253,4 126304 377462 /sbin/udevd
udevd 743 root mem REG 253,4 61622 1703961 /lib64/libnss_files-2.11.3.so
udevd 743 root mem REG 253,4 52424 1703965 /lib64/libnss_nis-2.11.3.so
udevd 743 root mem REG 253,4 108248 1703955 /lib64/libnsl-2.11.3.so
udevd 743 root mem REG 253,4 38676 1703957 /lib64/libnss_compat-2.11.3.so
udevd 743 root mem REG 253,4 19149 1703950 /lib64/libdl-2.11.3.so
udevd 743 root mem REG 253,4 1754140 1703944 /lib64/libc-2.11.3.so
udevd 743 root mem REG 253,4 118080 1704003 /lib64/libselinux.so.1
udevd 743 root mem REG 253,4 151051 1706614 /lib64/ld-2.11.3.so
edac-poll 905 root cwd DIR 253,4 4096 2 /
edac-poll 905 root rtd DIR 253,4 4096 2 /
kauditd 1163 root cwd DIR 253,4 4096 2 /
kauditd 1163 root rtd DIR 253,4 4096 2 /
kworker/0 1521 root cwd DIR 253,4 4096 2 /
kworker/0 1521 root rtd DIR 253,4 4096 2 /
kworker/1 2071 root cwd DIR 253,4 4096 2 /
kworker/1 2071 root rtd DIR 253,4 4096 2 /
udevd 4505 root cwd DIR 253,4 4096 2 /
udevd 4505 root rtd DIR 253,4 4096 2 /
udevd 4505 root txt REG 253,4 126304 377462 /sbin/udevd
udevd 4505 root mem REG 253,4 61622 1703961 /lib64/libnss_files-2.11.3.so
udevd 4505 root mem REG 253,4 52424 1703965 /lib64/libnss_nis-2.11.3.so
udevd 4505 root mem REG 253,4 108248 1703955 /lib64/libnsl-2.11.3.so
udevd 4505 root mem REG 253,4 38676 1703957 /lib64/libnss_compat-2.11.3.so
udevd 4505 root mem REG 253,4 19149 1703950 /lib64/libdl-2.11.3.so
udevd 4505 root mem REG 253,4 1754140 1703944 /lib64/libc-2.11.3.so
udevd 4505 root mem REG 253,4 118080 1704003 /lib64/libselinux.so.1
udevd 4505 root mem REG 253,4 151051 1706614 /lib64/ld-2.11.3.so
udevd 4506 root cwd DIR 253,4 4096 2 /
udevd 4506 root rtd DIR 253,4 4096 2 /
udevd 4506 root txt REG 253,4 126304 377462 /sbin/udevd
udevd 4506 root mem REG 253,4 61622 1703961 /lib64/libnss_files-2.11.3.so
udevd 4506 root mem REG 253,4 52424 1703965 /lib64/libnss_nis-2.11.3.so
udevd 4506 root mem REG 253,4 108248 1703955 /lib64/libnsl-2.11.3.so
udevd 4506 root mem REG 253,4 38676 1703957 /lib64/libnss_compat-2.11.3.so
udevd 4506 root mem REG 253,4 19149 1703950 /lib64/libdl-2.11.3.so
udevd 4506 root mem REG 253,4 1754140 1703944 /lib64/libc-2.11.3.so
udevd 4506 root mem REG 253,4 118080 1704003 /lib64/libselinux.so.1
udevd 4506 root mem REG 253,4 151051 1706614 /lib64/ld-2.11.3.so
pulseaudi 4810 gdm cwd DIR 253,4 4096 2 /
pulseaudi 4810 gdm rtd DIR 253,4 4096 2 /
pulseaudi 4810 gdm txt REG 253,4 75984 287361 /usr/bin/pulseaudio
pulseaudi 4810 gdm mem REG 253,4 14536 49767 /usr/lib64/pulse-0.9.23/modules/module-cork-music-on-phone.so
pulseaudi 4810 gdm mem REG 253,4 10456 49793 /usr/lib64/pulse-0.9.23/modules/module-position-event-sounds.so
pulseaudi 4810 gdm mem REG 253,4 18856 49766 /usr/lib64/pulse-0.9.23/modules/module-console-kit.so
pulseaudi 4810 gdm mem REG 253,4 22880 49804 /usr/lib64/pulse-0.9.23/modules/module-suspend-on-idle.so
pulseaudi 4810 gdm mem REG 253,4 14616 49781 /usr/lib64/pulse-0.9.23/modules/module-intended-roles.so
pulseaudi 4810 gdm mem REG 253,4 18944 49789 /usr/lib64/pulse-0.9.23/modules/module-null-sink.so
pulseaudi 4810 gdm mem REG 253,4 10488 49759 /usr/lib64/pulse-0.9.23/modules/module-always-sink.so
pulseaudi 4810 gdm mem REG 253,4 14544 49795 /usr/lib64/pulse-0.9.23/modules/module-rescue-streams.so
pulseaudi 4810 gdm mem REG 253,4 14576 49768 /usr/lib64/pulse-0.9.23/modules/module-default-device-restore.so
pulseaudi 4810 gdm mem REG 253,4 14672 49788 /usr/lib64/pulse-0.9.23/modules/module-native-protocol-unix.so
pulseaudi 4810 gdm mem REG 253,4 52792 49751 /usr/lib64/pulse-0.9.23/modules/libprotocol-esound.so
pulseaudi 4810 gdm mem REG 253,4 14656 49776 /usr/lib64/pulse-0.9.23/modules/module-esound-protocol-unix.so
pulseaudi 4810 gdm mem REG 253,4 56056 1706420 /lib64/libudev.so.0.5.1
pulseaudi 4810 gdm mem REG 253,4 27280 49807 /usr/lib64/pulse-0.9.23/modules/module-udev-detect.so
pulseaudi 4810 gdm mem REG 253,4 19008 49760 /usr/lib64/pulse-0.9.23/modules/module-augment-properties.so
pulseaudi 4810 gdm mem REG 253,4 14632 49761 /usr/lib64/pulse-0.9.23/modules/module-card-restore.so
pulseaudi 4810 gdm mem REG 253,4 127112 49753 /usr/lib64/pulse-0.9.23/modules/libprotocol-native.so
pulseaudi 4810 gdm mem REG 253,4 35616 49803 /usr/lib64/pulse-0.9.23/modules/module-stream-restore.so
pulseaudi 4810 gdm mem REG 253,4 23024 49771 /usr/lib64/pulse-0.9.23/modules/module-device-restore.so
pulseaudi 4810 gdm mem REG 253,4 198280 1664437 /usr/lib64/libvorbis.so.0.4.0
pulseaudi 4810 gdm mem REG 253,4 22816 1664255 /usr/lib64/libogg.so.0.5.3
pulseaudi 4810 gdm mem REG 253,4 1947904 1664439 /usr/lib64/libvorbisenc.so.2.0.3
pulseaudi 4810 gdm mem REG 253,4 312376 1664731 /usr/lib64/libFLAC.so.8.2.0
pulseaudi 4810 gdm mem REG 253,4 1754140 1703944 /lib64/libc-2.11.3.so
pulseaudi 4810 gdm mem REG 253,4 135690 1703970 /lib64/libpthread-2.11.3.so
pulseaudi 4810 gdm mem REG 253,4 541821 1703952 /lib64/libm-2.11.3.so
pulseaudi 4810 gdm mem REG 253,4 19149 1703950 /lib64/libdl-2.11.3.so
pulseaudi 4810 gdm mem REG 253,4 47241 1703974 /lib64/librt-2.11.3.so
pulseaudi 4810 gdm mem REG 253,4 22952 1664331 /usr/lib64/libgdbm.so.3.0.0
pulseaudi 4810 gdm mem REG 253,4 253488 1706423 /lib64/libdbus-1.so.3.4.0
pulseaudi 4810 gdm mem REG 253,4 385176 1664921 /usr/lib64/libsndfile.so.1.0.20
pulseaudi 4810 gdm mem REG 253,4 81768 1664237 /usr/lib64/libspeexdsp.so.1.5.0
pulseaudi 4810 gdm mem REG 253,4 1480912 1664239 /usr/lib64/libsamplerate.so.0.1.4
pulseaudi 4810 gdm mem REG 253,4 39600 1664286 /usr/lib64/libltdl.so.7.2.0
pulseaudi 4810 gdm mem REG 253,4 42016 1703984 /lib64/libwrap.so.0.7.6
pulseaudi 4810 gdm mem REG 253,4 14592 1664176 /usr/lib64/libXau.so.6.0.0
pulseaudi 4810 gdm mem REG 253,4 113520 1664410 /usr/lib64/libxcb.so.1.0.0
pulseaudi 4810 gdm mem REG 253,4 14472 377531 /usr/lib64/libxcb-atom.so.0.0.0
pulseaudi 4810 gdm mem REG 253,4 6224 1664400 /usr/lib64/libxcb-xlib.so.0.0.0
pulseaudi 4810 gdm mem REG 253,4 1298176 1665721 /usr/lib64/libX11.so.6.2.0
pulseaudi 4810 gdm mem REG 253,4 73624 1666222 /usr/lib64/libXext.so.6.4.0
pulseaudi 4810 gdm mem REG 253,4 27224 377513 /usr/lib64/libXtst.so.6.1.0
pulseaudi 4810 gdm mem REG 253,4 102544 1664178 /usr/lib64/libICE.so.6.3.0
pulseaudi 4810 gdm mem REG 253,4 19152 1704053 /lib64/libuuid.so.1.3.0
pulseaudi 4810 gdm mem REG 253,4 35136 1664841 /usr/lib64/libSM.so.6.0.0
pulseaudi 4810 gdm mem REG 253,4 6112 1665719 /usr/lib64/libX11-xcb.so.1.0.0
pulseaudi 4810 gdm mem REG 253,4 325928 377771 /usr/lib64/libpulsecommon-0.9.23.so
pulseaudi 4810 gdm mem REG 253,4 275896 377770 /usr/lib64/libpulse.so.0.12.4
pulseaudi 4810 gdm mem REG 253,4 531864 378054 /usr/lib64/libpulsecore-0.9.23.so
pulseaudi 4810 gdm mem REG 253,4 151051 1706614 /lib64/ld-2.11.3.so
pulseaudi 4810 gdm mem REG 253,4 123193 1600576 /usr/share/locale/de/LC_MESSAGES/libc.mo
pulseaudi 4810 gdm mem REG 253,4 256484 1771347 /usr/lib/locale/de_DE.utf8/LC_CTYPE
pulseaudi 4810 gdm mem REG 253,4 1163682 1771451 /usr/lib/locale/de_DE.utf8/LC_COLLATE
pulseaudi 4810 gdm mem REG 253,4 38658 1601843 /usr/share/locale/de/LC_MESSAGES/pulseaudio.mo
pulseaudi 4810 gdm mem REG 253,4 54 1778992 /usr/lib/locale/de_DE.utf8/LC_NUMERIC
pulseaudi 4810 gdm mem REG 253,4 2354 1771453 /usr/lib/locale/de_DE.utf8/LC_TIME
pulseaudi 4810 gdm mem REG 253,4 294 1786318 /usr/lib/locale/de_DE.utf8/LC_MONETARY
pulseaudi 4810 gdm mem REG 253,4 54 1778947 /usr/lib/locale/de_DE.utf8/LC_MESSAGES/SYS_LC_MESSAGES
pulseaudi 4810 gdm mem REG 253,4 34 1786329 /usr/lib/locale/de_DE.utf8/LC_PAPER
pulseaudi 4810 gdm mem REG 253,4 83 1779070 /usr/lib/locale/de_DE.utf8/LC_NAME
pulseaudi 4810 gdm mem REG 253,4 155 1770366 /usr/lib/locale/de_DE.utf8/LC_ADDRESS
pulseaudi 4810 gdm mem REG 253,4 56 1778940 /usr/lib/locale/de_DE.utf8/LC_TELEPHONE
pulseaudi 4810 gdm mem REG 253,4 23 1788036 /usr/lib/locale/de_DE.utf8/LC_MEASUREMENT
pulseaudi 4810 gdm mem REG 253,4 375 1771452 /usr/lib/locale/de_DE.utf8/LC_IDENTIFICATION
pulseaudi 4810 gdm mem REG 253,4 26050 1786333 /usr/lib64/gconv/gconv-modules.cache
pulseaudi 4810 gdm 10u REG 253,4 3898 385662 /var/lib/gdm/.pulse/2cbcb5ea89b045bfd1660cb04fb22f8f-device-volumes.x86_64-suse-linux-gnu.gdbm
pulseaudi 4810 gdm 11u REG 253,4 3072 385663 /var/lib/gdm/.pulse/2cbcb5ea89b045bfd1660cb04fb22f8f-stream-volumes.x86_64-suse-linux-gnu.gdbm
pulseaudi 4810 gdm 12u REG 253,4 3072 385664 /var/lib/gdm/.pulse/2cbcb5ea89b045bfd1660cb04fb22f8f-card-database.x86_64-suse-linux-gnu.gdbm
sshd 4838 root cwd DIR 253,4 4096 2 /
sshd 4838 root rtd DIR 253,4 4096 2 /
sshd 4838 root txt REG 253,4 454768 1666561 /usr/sbin/sshd
sshd 4838 root mem REG 253,4 217016 369452 /var/run/nscd/group
sshd 4838 root mem REG 253,4 6104 1722560 /lib64/security/pam_deny.so
sshd 4838 root mem REG 253,4 6192 1722597 /lib64/security/pam_warn.so
sshd 4838 root mem REG 253,4 10456 1722595 /lib64/security/pam_umask.so
sshd 4838 root mem REG 253,4 18832 1722571 /lib64/security/pam_limits.so
sshd 4838 root mem REG 253,4 10392 1722574 /lib64/security/pam_loginuid.so
sshd 4838 root mem REG 253,4 39496 1664784 /usr/lib64/libcrack.so.2.8.0
sshd 4838 root mem REG 253,4 23024 1722734 /lib64/security/pam_pwcheck.so
sshd 4838 root mem REG 253,4 22896 1703988 /lib64/libxcrypt.so.2.0.0
sshd 4838 root mem REG 253,4 52288 1722735 /lib64/security/pam_unix2.so
sshd 4838 root mem REG 253,4 14552 1722562 /lib64/security/pam_env.so
sshd 4838 root mem REG 253,4 10360 1722579 /lib64/security/pam_nologin.so

sshd 4838 root mem REG 253,4 61622 1703961 /lib64/libnss_files-2.11.3.so
sshd 4838 root mem REG 253,4 135690 1703970 /lib64/libpthread-2.11.3.so
sshd 4838 root mem REG 253,4 10336 1704032 /lib64/libkeyutils-1.2.so
sshd 4838 root mem REG 253,4 35800 1665941 /usr/lib64/libkrb5support.so.0.1
sshd 4838 root mem REG 253,4 22944 378061 /usr/lib64/libscconf.so.2.0.0
sshd 4838 root mem REG 253,4 39600 1664286 /usr/lib64/libltdl.so.7.2.0
sshd 4838 root mem REG 253,4 39824 1664253 /usr/lib64/libopenct.so.1.0.0
sshd 4838 root mem REG 253,4 343040 1666427 /usr/lib64/libssl.so.0.9.8
sshd 4838 root mem REG 253,4 110440 1666050 /usr/lib64/libsasl2.so.2.0.22
sshd 4838 root mem REG 253,4 1754140 1703944 /lib64/libc-2.11.3.so
sshd 4838 root mem REG 253,4 14720 1704064 /lib64/libcom_err.so.2.1
sshd 4838 root mem REG 253,4 154776 1665929 /usr/lib64/libk5crypto.so.3.1
sshd 4838 root mem REG 253,4 664216 1665939 /usr/lib64/libkrb5.so.3.3
sshd 4838 root mem REG 253,4 185904 1665925 /usr/lib64/libgssapi_krb5.so.2.2
sshd 4838 root mem REG 253,4 98115 1703972 /lib64/libresolv-2.11.3.so
sshd 4838 root mem REG 253,4 661736 378057 /usr/lib64/libopensc.so.2.0.0
sshd 4838 root mem REG 253,4 57699 1703948 /lib64/libcrypt-2.11.3.so
sshd 4838 root mem REG 253,4 108248 1703955 /lib64/libnsl-2.11.3.so
sshd 4838 root mem REG 253,4 88704 1703982 /lib64/libz.so.1.2.3
sshd 4838 root mem REG 253,4 14360 1703978 /lib64/libutil-2.11.3.so
sshd 4838 root mem REG 253,4 60584 1666335 /usr/lib64/liblber-2.4.so.2.7.1
sshd 4838 root mem REG 253,4 298088 1666337 /usr/lib64/libldap-2.4.so.2.7.1
sshd 4838 root mem REG 253,4 1621368 1664713 /usr/lib64/libcrypto.so.0.9.8
sshd 4838 root mem REG 253,4 118080 1704003 /lib64/libselinux.so.1
sshd 4838 root mem REG 253,4 19149 1703950 /lib64/libdl-2.11.3.so
sshd 4838 root mem REG 253,4 56048 1706410 /lib64/libpam.so.0.83.1
sshd 4838 root mem REG 253,4 109128 1704036 /lib64/libaudit.so.0.0.0
sshd 4838 root mem REG 253,4 42016 1703984 /lib64/libwrap.so.0.7.6
sshd 4838 root mem REG 253,4 151051 1706614 /lib64/ld-2.11.3.so
sshd 4838 root DEL REG 253,4 369453 /var/run/nscd/dbyEJuqe
sshd 4838 root mem REG 253,4 217016 369451 /var/run/nscd/passwd
bash 4861 root cwd DIR 253,4 4096 860161 /tmp
bash 4861 root rtd DIR 253,4 4096 2 /
bash 4861 root txt REG 253,4 584016 1482762 /bin/bash
bash 4861 root mem REG 253,4 293936 1704013 /lib64/libncurses.so.5.6
bash 4861 root mem REG 253,4 1754140 1703944 /lib64/libc-2.11.3.so
bash 4861 root mem REG 253,4 19149 1703950 /lib64/libdl-2.11.3.so
bash 4861 root mem REG 253,4 263568 1704043 /lib64/libreadline.so.5.2
bash 4861 root mem REG 253,4 151051 1706614 /lib64/ld-2.11.3.so
bash 4861 root mem REG 253,4 256484 1771347 /usr/lib/locale/de_DE.utf8/LC_CTYPE
bash 4861 root mem REG 253,4 54 1778992 /usr/lib/locale/de_DE.utf8/LC_NUMERIC
bash 4861 root mem REG 253,4 2354 1771453 /usr/lib/locale/de_DE.utf8/LC_TIME
bash 4861 root mem REG 253,4 1163682 1771451 /usr/lib/locale/de_DE.utf8/LC_COLLATE
bash 4861 root mem REG 253,4 294 1786318 /usr/lib/locale/de_DE.utf8/LC_MONETARY
bash 4861 root mem REG 253,4 54 1778947 /usr/lib/locale/de_DE.utf8/LC_MESSAGES/SYS_LC_MESSAGES
bash 4861 root mem REG 253,4 34 1786329 /usr/lib/locale/de_DE.utf8/LC_PAPER
bash 4861 root mem REG 253,4 83 1779070 /usr/lib/locale/de_DE.utf8/LC_NAME
bash 4861 root mem REG 253,4 155 1770366 /usr/lib/locale/de_DE.utf8/LC_ADDRESS
bash 4861 root mem REG 253,4 56 1778940 /usr/lib/locale/de_DE.utf8/LC_TELEPHONE
bash 4861 root mem REG 253,4 23 1788036 /usr/lib/locale/de_DE.utf8/LC_MEASUREMENT
bash 4861 root mem REG 253,4 26050 1786333 /usr/lib64/gconv/gconv-modules.cache
bash 4861 root mem REG 253,4 375 1771452 /usr/lib/locale/de_DE.utf8/LC_IDENTIFICATION
bash 4861 root mem REG 253,4 217016 369451 /var/run/nscd/passwd
kworker/u 5107 root cwd DIR 253,4 4096 2 /
kworker/u 5107 root rtd DIR 253,4 4096 2 /
flush-253 5126 root cwd DIR 253,4 4096 2 /
flush-253 5126 root rtd DIR 253,4 4096 2 /
rc 5127 root cwd DIR 253,4 4096 2 /
rc 5127 root rtd DIR 253,4 4096 2 /
rc 5127 root txt REG 253,4 584016 1482762 /bin/bash
rc 5127 root mem REG 253,4 293936 1704013 /lib64/libncurses.so.5.6
rc 5127 root mem REG 253,4 1754140 1703944 /lib64/libc-2.11.3.so
rc 5127 root mem REG 253,4 19149 1703950 /lib64/libdl-2.11.3.so
rc 5127 root mem REG 253,4 263568 1704043 /lib64/libreadline.so.5.2
rc 5127 root mem REG 253,4 151051 1706614 /lib64/ld-2.11.3.so
rc 5127 root 255r REG 253,4 10161 1180634 /etc/init.d/rc
kworker/1 6969 root cwd DIR 253,4 4096 2 /
kworker/1 6969 root rtd DIR 253,4 4096 2 /
splash 7333 root cwd DIR 253,4 4096 2 /
splash 7333 root rtd DIR 253,4 4096 2 /
splash 7333 root txt REG 253,4 421680 378132 /sbin/splash
splash 7333 root mem REG 253,4 1754140 1703944 /lib64/libc-2.11.3.so
splash 7333 root mem REG 253,4 541821 1703952 /lib64/libm-2.11.3.so
splash 7333 root mem REG 253,4 88704 1703982 /lib64/libz.so.1.2.3
splash 7333 root mem REG 253,4 151051 1706614 /lib64/ld-2.11.3.so
S01reboot 7350 root cwd DIR 253,4 4096 2 /
S01reboot 7350 root rtd DIR 253,4 4096 2 /
S01reboot 7350 root txt REG 253,4 584016 1482762 /bin/bash
S01reboot 7350 root mem REG 253,4 293936 1704013 /lib64/libncurses.so.5.6
S01reboot 7350 root mem REG 253,4 1754140 1703944 /lib64/libc-2.11.3.so
S01reboot 7350 root mem REG 253,4 19149 1703950 /lib64/libdl-2.11.3.so
S01reboot 7350 root mem REG 253,4 263568 1704043 /lib64/libreadline.so.5.2
S01reboot 7350 root mem REG 253,4 151051 1706614 /lib64/ld-2.11.3.so
S01reboot 7350 root 255r REG 253,4 6255 1180631 /etc/init.d/halt
K07boot.m 7541 root cwd DIR 253,4 4096 2 /
K07boot.m 7541 root rtd DIR 253,4 4096 2 /
K07boot.m 7541 root txt REG 253,4 584016 1482762 /bin/bash
K07boot.m 7541 root mem REG 253,4 293936 1704013 /lib64/libncurses.so.5.6
K07boot.m 7541 root mem REG 253,4 1754140 1703944 /lib64/libc-2.11.3.so
K07boot.m 7541 root mem REG 253,4 19149 1703950 /lib64/libdl-2.11.3.so
K07boot.m 7541 root mem REG 253,4 263568 1704043 /lib64/libreadline.so.5.2
K07boot.m 7541 root mem REG 253,4 151051 1706614 /lib64/ld-2.11.3.so
K07boot.m 7541 root 255r REG 253,4 4320 1182081 /etc/init.d/boot.multipath
lsof 7548 root cwd DIR 253,4 4096 2 /
lsof 7548 root rtd DIR 253,4 4096 2 /
lsof 7548 root txt REG 253,4 127416 1664421 /usr/bin/lsof
lsof 7548 root mem REG 253,4 19149 1703950 /lib64/libdl-2.11.3.so
lsof 7548 root mem REG 253,4 1754140 1703944 /lib64/libc-2.11.3.so
lsof 7548 root mem REG 253,4 118080 1704003 /lib64/libselinux.so.1
lsof 7548 root mem REG 253,4 151051 1706614 /lib64/ld-2.11.3.so
lsof 7548 root 1w REG 253,4 0 860263 /tmp/lsof
lsof 7548 root 2w REG 253,4 0 860263 /tmp/lsof
lsof 7549 root cwd DIR 253,4 4096 2 /
lsof 7549 root rtd DIR 253,4 4096 2 /
lsof 7549 root txt REG 253,4 127416 1664421 /usr/bin/lsof
lsof 7549 root mem REG 253,4 19149 1703950 /lib64/libdl-2.11.3.so
lsof 7549 root mem REG 253,4 1754140 1703944 /lib64/libc-2.11.3.so
lsof 7549 root mem REG 253,4 118080 1704003 /lib64/libselinux.so.1
lsof 7549 root mem REG 253,4 151051 1706614 /lib64/ld-2.11.3.so




Another hint is to look to the order of the "K"-Scripts in /etc/init.d/boot.d.




<HOSTNAME>:/etc/init.d/boot.d # ls -al
total 8
drwxr-xr-x 2 root root 4096 May 22 13:33 .
drwxr-xr-x 11 root root 4096 May 23 09:42 ..
lrwxrwxrwx 1 root root 15 May 15 12:26 K01boot.cleanup -> ../boot.cleanup
lrwxrwxrwx 1 root root 18 May 15 12:29 K01boot.compliance -> ../boot.compliance
lrwxrwxrwx 1 root root 13 May 15 12:28 K01boot.cycle -> ../boot.cycle
lrwxrwxrwx 1 root root 15 May 15 12:26 K01boot.debugfs -> ../boot.debugfs
lrwxrwxrwx 1 root root 12 May 15 12:28 K01boot.fuse -> ../boot.fuse
lrwxrwxrwx 1 root root 16 May 15 12:26 K01boot.ipconfig -> ../boot.ipconfig
lrwxrwxrwx 1 root root 13 May 15 12:32 K01boot.kdump -> ../boot.kdump
lrwxrwxrwx 1 root root 12 May 15 12:26 K01boot.klog -> ../boot.klog
lrwxrwxrwx 1 root root 16 May 15 12:26 K01boot.ldconfig -> ../boot.ldconfig
lrwxrwxrwx 1 root root 19 May 15 12:28 K01boot.lvm_monitor -> ../boot.lvm_monitor
lrwxrwxrwx 1 root root 12 May 15 12:26 K01boot.proc -> ../boot.proc
lrwxrwxrwx 1 root root 13 May 15 12:27 K01boot.quota -> ../boot.quota
lrwxrwxrwx 1 root root 12 May 15 12:25 K01boot.scpm -> ../boot.scpm
lrwxrwxrwx 1 root root 14 May 15 12:25 K01boot.sysctl -> ../boot.sysctl
lrwxrwxrwx 1 root root 18 May 15 12:26 K01boot.udev_retry -> ../boot.udev_retry
lrwxrwxrwx 1 root root 13 May 15 12:26 K02boot.clock -> ../boot.clock
lrwxrwxrwx 1 root root 19 May 15 12:28 K02boot.loadmodules -> ../boot.loadmodules
lrwxrwxrwx 1 root root 16 May 15 12:30 K02boot.localnet -> ../boot.localnet
lrwxrwxrwx 1 root root 12 May 15 12:26 K02boot.swap -> ../boot.swap
lrwxrwxrwx 1 root root 15 May 15 12:28 K04boot.localfs -> ../boot.localfs
lrwxrwxrwx 1 root root 16 May 15 12:28 K06boot.rootfsck -> ../boot.rootfsck
lrwxrwxrwx 1 root root 17 May 15 13:18 K07boot.multipath -> ../boot.multipath
lrwxrwxrwx 1 root root 21 May 15 12:28 K08boot.device-mapper -> ../boot.device-mapper
lrwxrwxrwx 1 root root 12 May 15 12:28 K09boot.udev -> ../boot.udev
lrwxrwxrwx 1 root root 12 May 15 12:26 S01boot.udev -> ../boot.udev
lrwxrwxrwx 1 root root 16 May 15 12:26 S02boot.rootfsck -> ../boot.rootfsck
lrwxrwxrwx 1 root root 13 May 15 12:26 S03boot.clock -> ../boot.clock
lrwxrwxrwx 1 root root 21 May 15 12:27 S04boot.device-mapper -> ../boot.device-mapper
lrwxrwxrwx 1 root root 19 May 15 12:26 S04boot.loadmodules -> ../boot.loadmodules
lrwxrwxrwx 1 root root 17 May 15 13:18 S06boot.multipath -> ../boot.multipath
lrwxrwxrwx 1 root root 15 May 15 12:28 S10boot.localfs -> ../boot.localfs
lrwxrwxrwx 1 root root 18 May 15 12:29 S12boot.compliance -> ../boot.compliance
lrwxrwxrwx 1 root root 13 May 15 12:29 S13boot.cycle -> ../boot.cycle
lrwxrwxrwx 1 root root 12 May 15 12:29 S13boot.fuse -> ../boot.fuse
lrwxrwxrwx 1 root root 12 May 15 12:29 S13boot.klog -> ../boot.klog
lrwxrwxrwx 1 root root 19 May 15 12:29 S13boot.lvm_monitor -> ../boot.lvm_monitor
lrwxrwxrwx 1 root root 12 May 15 12:29 S13boot.proc -> ../boot.proc
lrwxrwxrwx 1 root root 12 May 15 12:29 S13boot.scpm -> ../boot.scpm
lrwxrwxrwx 1 root root 12 May 15 12:29 S13boot.swap -> ../boot.swap
lrwxrwxrwx 1 root root 18 May 15 12:29 S13boot.udev_retry -> ../boot.udev_retry
lrwxrwxrwx 1 root root 15 May 15 12:29 S14boot.debugfs -> ../boot.debugfs
lrwxrwxrwx 1 root root 16 May 15 12:29 S14boot.ldconfig -> ../boot.ldconfig
lrwxrwxrwx 1 root root 13 May 15 12:29 S14boot.quota -> ../boot.quota
lrwxrwxrwx 1 root root 14 May 15 12:29 S14boot.sysctl -> ../boot.sysctl
lrwxrwxrwx 1 root root 15 May 15 12:29 S15boot.cleanup -> ../boot.cleanup
lrwxrwxrwx 1 root root 16 May 15 12:29 S15boot.ipconfig -> ../boot.ipconfig
lrwxrwxrwx 1 root root 16 May 15 12:29 S16boot.localnet -> ../boot.localnet
lrwxrwxrwx 1 root root 13 May 15 12:32 S17boot.kdump -> ../boot.kdump





What's your output of 'chkconfig --list -A | grep -i multi'? Are you using the root-partition within LVM? Or as a normal primary-partition?

The Output is:


<HOSTNAME>:/etc/init.d/boot.d # chkconfig --list -A | grep -i multi
boot.multipath 0:off 1:off 2:off 3:off 4:off 5:off 6:off B:on
multipathd 0:off 1:off 2:off 3:on 4:off 5:on 6:off


All Partitions are created with YaST as normal Primary Partitions with EXT3 Filesystem.
No LVM is used.




Can you then also try to remark the line 'features "1 queue_if_no_path"' in your multipath.conf?
From "man multipath.conf":
Would you suggest using the option:



"no_path_retry <VALUE>"


instead of queueing the I/O load???

Thanks again!

amo_vzug
24-May-2012, 18:42
I was also able to reproduce this behavior (SAN-Boot with root-fs within lvm). But I did not have special fsck's after a shutdown/reboot. Because of this, my filesystems looks consistent. So for me, the "map in use"-error is poor, but it seems harmless.