PDA

View Full Version : DRBD modules missing after upgrade to SLES 15



johngoutbeck
14-Feb-2019, 07:33
Upgraded from SLES 12 SP4 + HA to SLES 15 + HA
Using DRBD on devices & HA cluster to have HA on the DRBD devices

after the upgrade, DRBD will not start

ISSUE
nss-sn01:/ # drbdadm status
modinfo: ERROR: Module drbd not found.
modinfo: ERROR: Module drbd not found.
modprobe: FATAL: Module drbd not found in directory /lib/modules/4.12.14-95.3-default
Failed to modprobe drbd (No such file or directory)
Command 'drbdsetup status' terminated with exit code 20
command status exited with code 20

---


nss-sn01:/ # rpm -qa | grep drbd
yast2-drbd-4.0.3-3.3.1.noarch
drbd-9.0.13+git.b83ade31-3.2.1.x86_64
drbd-utils-9.3.1-1.21.x86_64
drbd-kmp-default-9.0.13+git.b83ade31_k4.12.14_23-3.2.1.x86_64

nss-sn01:/ # systemctl status drbd -l
‚óŹ drbd.service - DRBD -- please disable. Unless you are NOT using a cluster manager.
Loaded: loaded (/usr/lib/systemd/system/drbd.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Wed 2019-02-13 22:05:13 MST; 13min ago
Main PID: 13357 (code=exited, status=5)

Feb 13 22:05:13 nss-sn01 systemd[1]: Starting DRBD -- please disable. Unless you are NOT using a cluster manager....
Feb 13 22:05:13 nss-sn01 drbd[13357]: /lib/drbd/drbd: line 58: local: can only be used in a function
Feb 13 22:05:13 nss-sn01 modprobe[13370]: FATAL: Module drbd not found in directory /lib/modules/4.12.14-95.3-default
Feb 13 22:05:13 nss-sn01 drbd[13357]: Can not load the drbd module.
Feb 13 22:05:13 nss-sn01 systemd[1]: drbd.service: Main process exited, code=exited, status=5/NOTINSTALLED
Feb 13 22:05:13 nss-sn01 systemd[1]: Failed to start DRBD -- please disable. Unless you are NOT using a cluster manager..
Feb 13 22:05:13 nss-sn01 systemd[1]: drbd.service: Unit entered failed state.
Feb 13 22:05:13 nss-sn01 systemd[1]: drbd.service: Failed with result 'exit-code'.

nss-sn01:/ # uname -a
Linux nss-sn01 4.12.14-95.3-default #1 SMP Wed Dec 5 06:00:48 UTC 2018 (63a8d29) x86_64 x86_64 x86_64 GNU/Linux

nss-sn01:/ # cat /etc/os-release
NAME="SLES"
VERSION="15"
VERSION_ID="15"
PRETTY_NAME="SUSE Linux Enterprise Server 15"
ID="sles"
ID_LIKE="suse"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:15"

---
Don't know if this has anything to do with this, but some files are missing compared to a working SLES12 SP4 + HA
nss-sn01:/lib/modules/4.12.14-95.3-default # l weak-updates/updates/
total 4
drwxr-xr-x 1 root root 16 Feb 13 21:00 ./
drwxr-xr-x 1 root root 14 Dec 14 18:47 ../
lrwxrwxrwx 1 root root 51 Dec 14 18:47 crash.ko -> /lib/modules/4.12.14-94.41-default/updates/crash.ko
nss-sn01:/lib/modules/4.12.14-95.3-default #

--

used yast to updated the drbd modules - dd not help

Any comments, suggestion, fixes?

Automatic Reply
20-Feb-2019, 06:30
johngoutbeck,

It appears that in the past few days you have not received a response to your
posting. That concerns us, and has triggered this automated reply.

These forums are peer-to-peer, best effort, volunteer run and that if your issue
is urgent or not getting a response, you might try one of the following options:

- Visit http://www.suse.com/support and search the knowledgebase and/or check all
the other support options available.
- Open a service request: https://www.suse.com/support
- You could also try posting your message again. Make sure it is posted in the
correct newsgroup. (http://forums.suse.com)

Be sure to read the forum FAQ about what to expect in the way of responses:
http://forums.suse.com/faq.php

If this is a reply to a duplicate posting or otherwise posted in error, please
ignore and accept our apologies and rest assured we will issue a stern reprimand
to our posting bot..

Good luck!

Your SUSE Forums Team
http://forums.suse.com

jmozdzen
20-Feb-2019, 16:40
Hi,

> FATAL: Module drbd not found in directory /lib/modules/4.12.14-95.3-default

what does "rpm -ql drbd-kmp-default-9.0.13+git.b83ade31_k4.12.14_23-3.2.1.x86_64" report? That the file is placed in /lib/modules/4.12.14-23*?

I have not installed SLES15 + HAE yet, but with earlier versions, there was a separate kernel shipped via HAE. Could you please check via according calls to "zypper se -s -i drbd" and "zypper se -s -ikernel", that the packages come from the same (HAE) repository / channel?

What also caught my eye is the version mismatch 9.0.13 vs. 9.3.1:

- drbd-9.0.13+git.b83ade31-3.2.1.x86_64
- drbd-kmp-default-9.0.13+git.b83ade31_k4.12.14_23-3.2.1.x86_64

but

- drbd-utils-9.3.1-1.21.x86_64

Regards,
J

johngoutbeck
21-Feb-2019, 05:25
Thanks for looking at this issue jmozdzen

output of commands you asked for.

---

nss-sn01:~ # rpm -ql drbd-kmp-default-9.0.13+git.b83ade31_k4.12.14_23-3.2.1.x86_64
/lib/modules/4.12.14-23-default
/lib/modules/4.12.14-23-default/updates
/lib/modules/4.12.14-23-default/updates/drbd.ko
/lib/modules/4.12.14-23-default/updates/drbd_transport_tcp.ko
nss-sn01:~ # zypper se -s -i drbd
Loading repository data...
Reading installed packages...

S | Name | Type | Version | Arch | Repository
---+------------------+---------+---------------------------------------+--------+-------------------------
i+ | drbd | package | 9.0.13+git.b83ade31-3.2.1 | x86_64 | SLE-Product-HA15-Updates
i | drbd-kmp-default | package | 9.0.13+git.b83ade31_k4.12.14_23-3.2.1 | x86_64 | SLE-Product-HA15-Updates
i | drbd-utils | package | 9.3.1-1.21 | x86_64 | SLE-Product-HA15-Pool
i+ | yast2-drbd | package | 4.0.3-3.3.1 | noarch | SLE-Product-HA15-Updates

Note: For an extended search including not yet activated remote resources please use 'zypper
search-packages'.
nss-sn01:~ # zypper se -s -i kernel
Loading repository data...
Reading installed packages...

S | Name | Type | Version | Arch | Repository
---+-------------------+---------+-----------------+--------+--------------------------------
i+ | kernel-default | package | 4.12.14-95.3.1 | x86_64 | (System Packages)
i+ | kernel-default | package | 4.12.14-25.28.1 | x86_64 | SLE-Module-Basesystem15-Updates
i | kernel-firmware | package | 20190118-3.14.1 | noarch | SLE-Module-Basesystem15-Updates
i+ | nfs-kernel-server | package | 2.1.1-6.10.2 | x86_64 | SLE-Module-Basesystem15-Updates

Note: For an extended search including not yet activated remote resources please use 'zypper
search-packages'.
---

It all looks OK

I trust these help with the diagnoses and fix.

And yes I did see the drbd packages difference.

jmozdzen
21-Feb-2019, 14:49
Hi, John,

> S | Name | Type | Version | Arch | Repository
> ---+-------------------+---------+-----------------+--------+--------------------------------
> i+ | kernel-default | package | 4.12.14-95.3.1 | x86_64 | (System Packages)
> i+ | kernel-default | package | 4.12.14-25.28.1 | x86_64 | SLE-Module-Basesystem15-Updates

"(System packages)" usually points at packages that are not coming from (currently connected) repositories. Might it be you upgraded the kernel package from somewhere else, i.e. a temporarily attached reposirory?

If you're not depending on the 4.12.14-95.3.1 kernel, you might want to test downgrading to the 4.12.14-25.28.1 kernel as provided by the SLE-Module-Basesystem15-Updates repository - my hope is that the DRBD kernel module (9.0.13+git.b83ade31_k4.12.14_23-3.2.1) might work with that kernel.

If you want to have a look at where that .95 kernel caes from, you could beforehand run "rpm -qi kernel-default" and save that output somewhere.

Regards,
J

johngoutbeck
21-Feb-2019, 20:23
Thanks jmozdzen;

more info
nss-sn01:~ # rpm -qi kernel-default
Name : kernel-default
Version : 4.12.14
Release : 95.3.1
Architecture: x86_64
Install Date: Fri Dec 14 18:47:02 2018
Group : System/Kernel
Size : 244114512
License : GPL-2.0
Signature : RSA/SHA256, Wed Dec 5 09:04:31 2018, Key ID 70af9e8139db7c82
Source RPM : kernel-default-4.12.14-95.3.1.nosrc.rpm
Build Date : Wed Dec 5 08:48:24 2018
Build Host : sheep53
Relocations : (not relocatable)
Packager : https://www.suse.com/
Vendor : SUSE LLC <https://www.suse.com/>
URL : http://www.kernel.org/
Summary : The Standard Kernel
Description :
The standard kernel for both uniprocessor and multiprocessor systems.


Source Timestamp: 2018-12-05 06:00:48 +0000
GIT Revision: 63a8d29f03cd2829d519ffb9ea2e3c56a36d0674
GIT Branch: SLE12-SP4
Distribution: SUSE Linux Enterprise 12
Name : kernel-default
Version : 4.12.14
Release : 25.28.1
Architecture: x86_64
Install Date: Wed Feb 13 20:54:49 2019
Group : System/Kernel
Size : 230703117
License : GPL-2.0
Signature : RSA/SHA256, Fri Jan 18 00:25:36 2019, Key ID 70af9e8139db7c82
Source RPM : kernel-default-4.12.14-25.28.1.nosrc.rpm
Build Date : Fri Jan 18 00:21:27 2019
Build Host : sheep88
Relocations : (not relocatable)
Packager : https://www.suse.com/
Vendor : SUSE LLC <https://www.suse.com/>
URL : http://www.kernel.org/
Summary : The Standard Kernel
Description :
The standard kernel for both uniprocessor and multiprocessor systems.


Source Timestamp: 2019-01-16 20:00:47 +0000
GIT Revision: dd6077c8eed7e05c2c9c81f1ad5c239067ab7ecb
GIT Branch: SLE15
Distribution: SUSE Linux Enterprise 15
nss-sn01:~ #
--

Looking at the Grub2 menu, the older SLES 15 kernel was listed to boot from - menuentry 'SLES 15, with Linux 4.12.14-25.28-default (recovery mode)'.
So I rebooted the system and choose - submenu 'Advanced options for SLES 15', then the above entry.
Now the system boot and startd DRBD

nss-sn01:~ # drbdadm status
drbd01 role:Secondary
disk:UpToDate
nss-sn02 role:Primary
peer-disk:UpToDate

drbd02 role:Secondary
disk:UpToDate
nss-sn02 role:Primary
peer-disk:UpToDate

sbd01 role:Primary
disk:UpToDate
nss-sn02 role:Secondary
peer-disk:UpToDate

sbd02 role:Secondary
disk:UpToDate
nss-sn02 role:Primary
peer-disk:UpToDate

nss-sn01:~ # uname -a
Linux nss-sn01 4.12.14-25.28-default #1 SMP Wed Jan 16 20:00:47 UTC 2019 (dd6077c) x86_64 x86_64 x86_64 GNU/Linux
nss-sn01:~ #

--
Pacemaker started and put the resources on the proper node.

Seems to be working now, but not on the newest kernel.

Should I leave it this way? How to rebuild to the latest kernel with DRBD modules added.?

jmozdzen
22-Feb-2019, 19:03
Hi John,



nss-sn01:~ # rpm -qi kernel-default
Name : kernel-default
Version : 4.12.14
Release : 95.3.1
Architecture: x86_64
Install Date: Fri Dec 14 18:47:02 2018
Group : System/Kernel
Size : 244114512
License : GPL-2.0
Signature : RSA/SHA256, Wed Dec 5 09:04:31 2018, Key ID 70af9e8139db7c82
Source RPM : kernel-default-4.12.14-95.3.1.nosrc.rpm
Build Date : Wed Dec 5 08:48:24 2018
Build Host : sheep53
Relocations : (not relocatable)
Packager : https://www.suse.com/
Vendor : SUSE LLC <https://www.suse.com/>
URL : http://www.kernel.org/
Summary : The Standard Kernel
Description :
The standard kernel for both uniprocessor and multiprocessor systems.


Source Timestamp: 2018-12-05 06:00:48 +0000
GIT Revision: 63a8d29f03cd2829d519ffb9ea2e3c56a36d0674
GIT Branch: SLE12-SP4
Distribution: SUSE Linux Enterprise 12


[extra lines inserted for visual clarity]


Name : kernel-default
Version : 4.12.14
Release : 25.28.1
Architecture: x86_64
Install Date: Wed Feb 13 20:54:49 2019
Group : System/Kernel
Size : 230703117
License : GPL-2.0
Signature : RSA/SHA256, Fri Jan 18 00:25:36 2019, Key ID 70af9e8139db7c82
Source RPM : kernel-default-4.12.14-25.28.1.nosrc.rpm
Build Date : Fri Jan 18 00:21:27 2019
Build Host : sheep88
Relocations : (not relocatable)
Packager : https://www.suse.com/
Vendor : SUSE LLC <https://www.suse.com/>
URL : http://www.kernel.org/
Summary : The Standard Kernel
Description :
The standard kernel for both uniprocessor and multiprocessor systems.


Source Timestamp: 2019-01-16 20:00:47 +0000
GIT Revision: dd6077c8eed7e05c2c9c81f1ad5c239067ab7ecb
GIT Branch: SLE15
Distribution: SUSE Linux Enterprise 15
nss-sn01:~ #

looking at the install dates of these packages and the "Distribution" info, you can see that the 4.12.14-95.3.1 kernel is from the old SLES12 system. With other words, it's not "the latest kernel" with regards to your SLES15 channels.

I recommend to remove that old kernel from your system, and a call to "rpm -qa|grep 95.3.1" might reveal some other packages related to that kernel which can also be deleted. "rpm -qa |grep 'System packages' " might list more orphans from SLES12, which then will have to be looked at individually - it's you to decide what to do with them :-/

Regards,
J

johngoutbeck
21-Jun-2019, 22:18
I should have replied sooner.

I wiped out the boot HD and clean install SLES 15 + HA extension.

All works well now.