# stopping pacemaker doesnt move resources to other node
**URL:** https://forums.suse.com/t/stopping-pacemaker-doesnt-move-resources-to-other-node/34131
**Category:** SLES High Availability Extension
**Created:** [June 26, 2019, 4:12pm UTC](https://forums.suse.com/t/stopping-pacemaker-doesnt-move-resources-to-other-node/34131 "2019-06-26T16:12:29Z")
**Posts on this page:** 16
**Page:** 1
### Author:  [@mnshsnghl](https://forums.suse.com/u/mnshsnghl)
#### Post date: [June 26, 2019, 4:12pm UTC](https://forums.suse.com/t/stopping-pacemaker-doesnt-move-resources-to-other-node/34131/1 "2019-06-26T16:12:29Z")
I have created a 2 node cluster on sles12. Following is the configuration -
```
msnode1:~ # crm --version
crm 3.0.0
msnode1:~ # corosync -v
Corosync Cluster Engine, version '2.3.6'
Copyright (c) 2006-2009 Red Hat, Inc.
```
msnode1:~ # crm config show
node 1: msnode1
node 2: msnode2
primitive mspersonal systemd:mspersonal \
op monitor interval=30s
primitive virtip IPaddr \
params ip=10.243.109.103 cidr\_netmask=21 \
op monitor interval=30s
location cli-prefer-virtip virtip role=Started inf: msnode1
colocation msconstraint inf: virtip mspersonal
order msorder Mandatory: virtip mspersonal
property cib-bootstrap-options: \
have-watchdog=false \
dc-version=1.1.16-4.8-77ea74d \
cluster-infrastructure=corosync \
cluster-name=mscluster \
stonith-enabled=false \
placement-strategy=balanced \
help \
list \
last-lrm-refresh=1561341732
rsc\_defaults rsc-options: \
resource-stickiness=100 \
migration-threshold=2
op\_defaults op-options: \
timeout=600 \
record-pending=true
msnode1:~ # crm status
Stack: corosync
Current DC: msnode1 (version 1.1.16-4.8-77ea74d) - partition with quorum
Last updated: Tue Jun 25 17:43:44 2019
Last change: Tue Jun 25 17:38:21 2019 by hacluster via cibadmin on msnode1
2 nodes configured
2 resources configured
Online: [msnode1 msnode2]
Full list of resources:
virtip (ocf:đIPaddr): Started msnode1
mspersonal (systemd:mspersonal): Started msnode1
When I shut off the cluster on node1 (or reboot node1), the resources started on msnode2, but then they immediately turned off and situation changes as -
msnode1:~ # systemctl stop pacemaker
msnode2:~ # crm status
Stack: corosync
Current DC: msnode2 (version 1.1.16-4.8-77ea74d) - partition WITHOUT quorum
Last updated: Tue Jun 25 17:44:26 2019
Last change: Tue Jun 25 17:38:20 2019 by hacluster via cibadmin on msnode1
2 nodes configured
2 resources configured
Online: [msnode2]
OFFLINE: [msnode1]
Full list of resources:
virtip (ocf:đIPaddr): Stopped
mspersonal (systemd:mspersonal): Stopped
When I restart the pacemaker service on msnode1, the resources starts back on msnode1 â
msnode1:~ # systemctl start pacemaker
msnode1:~ # crm status
Stack: corosync
Current DC: msnode2 (version 1.1.16-4.8-77ea74d) - partition with quorum
Last updated: Tue Jun 25 17:46:09 2019
Last change: Tue Jun 25 17:38:20 2019 by hacluster via cibadmin on msnode1
2 nodes configured
2 resources configured
Online: [msnode1 msnode2]
Full list of resources:
virtip (ocf:đIPaddr): Started msnode1
mspersonal (systemd:mspersonal): Started msnode1
But when I redo the same exercise, the resources actually start on msnode2 -
msnode1:~ # systemctl stop pacemaker
msnode2:~ # crm status
Stack: corosync
Current DC: msnode2 (version 1.1.16-4.8-77ea74d) - partition WITHOUT quorum
Last updated: Tue Jun 25 17:47:00 2019
Last change: Tue Jun 25 17:38:20 2019 by hacluster via cibadmin on msnode1
2 nodes configured
2 resources configured
Online: [msnode2]
OFFLINE: [msnode1]
Full list of resources:
virtip (ocf:đIPaddr): Started msnode2
mspersonal (systemd:mspersonal): Started msnode2
But when I start pacemaker again on msnode1, the resources move back to msnode1 which I didnt expect because of stickiness set to 100 which works fine when any one of the resource fails.
I am not able to catch what I am missing in this cluster configuration, which can set everything right for me.
---
### Author:  [@strahil-nikolov-dxc](https://forums.suse.com/u/strahil-nikolov-dxc)
#### Post date: [June 26, 2019, 5:29pm UTC](https://forums.suse.com/t/stopping-pacemaker-doesnt-move-resources-to-other-node/34131/2 "2019-06-26T17:29:35Z")
> [@](#):
>
> location cli-prefer-virtip virtip role=Started inf: msnode1
this means that you have run the âcrm resource migrate virtip msnode1â. When node1 dies - it will migrate to node2 and then when node1 is back - it will recover on node1.
Remove it by using "crm resource unmigrate virtip " and it will remove the cli-prefer.
Also, consider adding fencing mechanism - either SBD or any other type in order to guarantee that resources will be failed over.
Another issue:
> [@](#):
>
> Current DC: msnode2 (version 1.1.16-4.8-77ea74d) - partition WITHOUT quorum
you need the "two\_node:1 " set on both nodes (use csync2 -m /etc/corosync/corosync.conf ; csync2 -x to sync it between nodes)
Note , as the two\_node is enabled it will automatically enable âwait\_for\_allâ which means that if both nodes die (power outage , double fencing anything outstanding), then both nodes should be started before any resource is taken.
I would recommend it to leave it by default ,as you donât have a fencing mechanism and you could have the IP and the service started on both nodes - which is bad.
---
### Author:  [@mnshsnghl](https://forums.suse.com/u/mnshsnghl)
#### Post date: [June 26, 2019, 6:19pm UTC](https://forums.suse.com/t/stopping-pacemaker-doesnt-move-resources-to-other-node/34131/3 "2019-06-26T18:19:36Z")
[QUOTE=strahil-nikolov-dxc;58016]this means that you have run the âcrm resource migrate virtip msnode1â. When node1 dies - it will migrate to node2 and then when node1 is back - it will recover on node1.
> [@](#):
>
> Remove it by using "crm resource unmigrate virtip " and it will remove the cli-prefer.[/QUOTE]
>
> Thanks, I think this solved the problem. I am not sure how this got added, I didnt add it and didnt notice also.
>
> [QUOTE]Also, consider adding fencing mechanism - either SBD or any other type in order to guarantee that resources will be failed over.[/CODE][/CODE]
I am actually interested in adding a fencing mechanism, and would like to find out what is the other type of fencing that can be added and how ? And also, what is the best fencing mechanism. From my requirement perspective, I dont want a node to be rebooted as part of the fencing, is that possible ?
Another issue:
you need the "two\_node:1 " set on both nodes (use csync2 -m /etc/corosync/corosync.conf ; csync2 -x to sync it between nodes)
Note , as the two\_node is enabled it will automatically enable âwait\_for\_allâ which means that if both nodes die (power outage , double fencing anything outstanding), then both nodes should be started before any resource is taken.
I would recommend it to leave it by default ,as you donât have a fencing mechanism and you could have the IP and the service started on both nodes - which is bad.[/QUOTE]
I probably didnt understand this feature. You mean if I have two\_node=0, then there are greater chances of split brain ? and if two\_node=1, then both nodes should be up before resources are taken ? Well, I dont want both. I actually want the node which comes up first with the cluster should take the resource, and other remain as standby.
However, if I setup fencing, will this attribute required to be set ? Moreover, my cluster can grow and shrink dynamically (increase/decrease nodes), so dont want to rely on any such attribute.
Thanks a lot!
---
### Author:  [@strahil-nikolov-dxc](https://forums.suse.com/u/strahil-nikolov-dxc)
#### Post date: [June 26, 2019, 7:06pm UTC](https://forums.suse.com/t/stopping-pacemaker-doesnt-move-resources-to-other-node/34131/4 "2019-06-26T19:06:47Z")
By default quorum is 50% + 1Vote and with 2 nodes to have quorum you need 2 votes.
Yet in a two node cluster - you want to have quorum even with 1 node, so there is a special option âtwo\_node:1â which tells that (we need only 1 vote to survive) to pacemaker.
So ,in your case you need the âtwo\_node:1â and set a proper stonith device (no matter sbd or other). Otherwise - this will never be a cluster,as it will be susceptible to split-brains.
If you add a 3rd node via âha-cluster-joinâ that flag (two\_node) is automatically removed and should be added when you use âha-cluster-removeâ to reduce from 3 nodes to 2.
---
### Author:  [@singhm16](https://forums.suse.com/u/singhm16)
#### Post date: [June 27, 2019, 12:05pm UTC](https://forums.suse.com/t/stopping-pacemaker-doesnt-move-resources-to-other-node/34131/5 "2019-06-27T12:05:35Z")
[QUOTE=strahil-nikolov-dxc;58019]By default quorum is 50% + 1Vote and with 2 nodes to have quorum you need 2 votes.
Yet in a two node cluster - you want to have quorum even with 1 node, so there is a special option âtwo\_node:1â which tells that (we need only 1 vote to survive) to pacemaker.[/QUOTE]
Understood. Thanks. However, I noticed that when I enabled this option, and when the node where the services are running reboots, the services and virtual IP dont migrate to other node which is running. Is that how it is supposed to work ?
> [@](#):
>
> set a proper stonith device (no matter sbd or other).
Can you please suggest a fencing mechanism where there is no compulsion to bring a node down but only bring down required services.
---
### Author:  [@mnshsnghl](https://forums.suse.com/u/mnshsnghl)
#### Post date: [June 27, 2019, 12:38pm UTC](https://forums.suse.com/t/stopping-pacemaker-doesnt-move-resources-to-other-node/34131/6 "2019-06-27T12:38:46Z")
[QUOTE]By default quorum is 50% + 1Vote and with 2 nodes to have quorum you need 2 votes.
Yet in a two node cluster - you want to have quorum even with 1 node, so there is a special option âtwo\_node:1â which tells that (we need only 1 vote to survive) to pacemaker.[/QUOTE]
Thanks for the explanation. Understood the point.
So ,in your case you need the âtwo\_node:1â and [QUOTE]set a proper stonith device (no matter sbd or other)[/QUOTE].
Can you please help me implement the fencing mechanism which doesnt shoot down the node but just stops the services that I want.
---
### Author:  [@strahil-nikolov-dxc](https://forums.suse.com/u/strahil-nikolov-dxc)
#### Post date: [June 28, 2019, 8:09pm UTC](https://forums.suse.com/t/stopping-pacemaker-doesnt-move-resources-to-other-node/34131/7 "2019-06-28T20:09:18Z")
[QUOTE=singhm16;58031]Understood. Thanks. However, I noticed that when I enabled this option, and when the node where the services are running reboots, the services and virtual IP dont migrate to other node which is running. Is that how it is supposed to work ?
Can you please suggest a fencing mechanism where there is no compulsion to bring a node down but only bring down required services.[/QUOTE]
Yes, when you reboot a node (or the service is being restarted) the cluster will try to minimize downtime and bring it up on the other node.
Of course that can be controlled via a location constraint order (for example if you have a 3 node cluster and you donât want the cluster to start the resource/group on a specific node).
About the fencing - the idea behind the STONITH (Shoot the other node in the head) is to guarantee that the resource will not be used by the problematic node (for example the node might freeze, but after a minute or two /in recent case 5 min/ it getâs recovered just to write something to the Filesystem) , so the cluster can safely start it up on the working node.
If your service is using a shared storage - you can use fence\_scsi (pcmk\_reboot\_action=âoffâ) ,which uses persistent reservations (requires the storage to support it - most does) , but there is no guarantee that the frozen node will release the IP if they are not in a single resource group.
So you should have :
1. Filesystem
2. IP
3. APP
Once the node is fenced - it will fail to write to filesystem and the resource will be marked as failed. Everything behind it will be stopped.
Edit:
Of course, setting the cluster in maintenance is mandatory when making system update (with sbd I would recommend you to stop the cluster software locally after setting the maintenance) or maintenance on the application itself.
---
### Author:  [@strahil-nikolov-dxc](https://forums.suse.com/u/strahil-nikolov-dxc)
#### Post date: [June 28, 2019, 8:16pm UTC](https://forums.suse.com/t/stopping-pacemaker-doesnt-move-resources-to-other-node/34131/8 "2019-06-28T20:16:36Z")
[QUOTE=mnshsnghl;58032]
Can you please help me implement the fencing mechanism which doesnt shoot down the node but just stops the services that I want.[/QUOTE]
Thatâs tricky. If you use shared storage , you can use fence\_scsi with pcmk\_reboot\_action=âoffâ which will deregister the system from the shared storage.
In order to work properly, you need to put the IP after the file system resource in a single group. Once, the node is fenced - the filesystem will not be available and the node will stop everything after the filesystem resource (for example Shared IP + App).
Edit: Check my previous comment for an example.
---
### Author:  [@strahil-nikolov-dxc](https://forums.suse.com/u/strahil-nikolov-dxc)
#### Post date: [June 28, 2019, 8:21pm UTC](https://forums.suse.com/t/stopping-pacemaker-doesnt-move-resources-to-other-node/34131/9 "2019-06-28T20:21:44Z")
For Filesystem monitoring use âOCF\_CHECK\_LEVEL=20â in the âopâ section .
---
### Author:  [@mnshsnghl](https://forums.suse.com/u/mnshsnghl)
#### Post date: [September 12, 2019, 2:48pm UTC](https://forums.suse.com/t/stopping-pacemaker-doesnt-move-resources-to-other-node/34131/10 "2019-09-12T14:48:48Z")
[QUOTE=strahil-nikolov-dxc;58040]Thatâs tricky. If you use shared storage , you can use fence\_scsi with pcmk\_reboot\_action=âoffâ which will deregister the system from the shared storage.
In order to work properly, you need to put the IP after the file system resource in a single group. Once, the node is fenced - the filesystem will not be available and the node will stop everything after the filesystem resource (for example Shared IP + App).
Edit: Check my previous comment for an example.[/QUOTE]
Hi,
Was trying to implement your suggestion but found that my SLES system (SLES15) doesnt have fence\_scsi resource agent at all -
$ crm configre ra list stonith
apcmaster apcmastersnmp apcsmart
baytech cyclades drac3
external/drac5 external/dracmc-telnet external/ec2
external/hetzner external/hmchttp external/ibmrsa
external/ibmrsa-telnet external/ipmi external/ippower9258
external/kdumpcheck external/libvirt external/nut
external/rackpdu external/riloe external/vcenter
external/vmware external/xen0 external/xen0-ha
fence\_legacy ibmhmc meatware
nw\_rpc100s rcd\_serial rps10
suicide wti\_mpc wti\_nps
Is there a way I can get this resource agent, or has it been replaced with something else ?
---
### Author:  [@strahil-nikolov-dxc](https://forums.suse.com/u/strahil-nikolov-dxc)
#### Post date: [September 12, 2019, 5:33pm UTC](https://forums.suse.com/t/stopping-pacemaker-doesnt-move-resources-to-other-node/34131/11 "2019-09-12T17:33:53Z")
I have it on my test SLES 15.1 cluster
[CODE]# crm ra list stonith | grep scsi
fence\_pve fence\_raritan fence\_rcd\_serial fence\_rhevm fence\_rsa fence\_rsb fence\_sanbox2 fence\_sbd fence\_scsi
# rpm -qf /usr/sbin/fence\_scsi
fence-agents-4.2.1+git.1537269352.7b1fd536-5.38.x86\_64[/CODE]
My patterns that I have selected during install
`S | Name | Summary | Type
---+---------------+----------------------------+--------
i | apparmor | AppArmor | pattern
i | base | Minimal Base System | pattern
i+ | enhanced_base | Enhanced Base System | pattern
i+ | ha_sles | High Availability | pattern
i+ | minimal_base | Minimal Appliance Base | pattern
i | yast2_basis | YaST System Administration | pattern
`
---
### Author:  [@mnshsnghl](https://forums.suse.com/u/mnshsnghl)
#### Post date: [September 12, 2019, 6:04pm UTC](https://forums.suse.com/t/stopping-pacemaker-doesnt-move-resources-to-other-node/34131/12 "2019-09-12T18:04:51Z")
[QUOTE=strahil-nikolov-dxc;58432]I have it on my test SLES 15.1 cluster
[CODE]# crm ra list stonith | grep scsi
fence\_pve fence\_raritan fence\_rcd\_serial fence\_rhevm fence\_rsa fence\_rsb fence\_sanbox2 fence\_sbd fence\_scsi
# rpm -qf /usr/sbin/fence\_scsi
fence-agents-4.2.1+git.1537269352.7b1fd536-5.38.x86\_64[/CODE]
My patterns that I have selected during install
`S | Name | Summary | Type
---+---------------+----------------------------+--------
i | apparmor | AppArmor | pattern
i | base | Minimal Base System | pattern
i+ | enhanced_base | Enhanced Base System | pattern
i+ | ha_sles | High Availability | pattern
i+ | minimal_base | Minimal Appliance Base | pattern
i | yast2_basis | YaST System Administration | pattern
`[/QUOTE]
Thanks. I think that will work. Let me try that.
---
### Author:  [@mnshsnghl](https://forums.suse.com/u/mnshsnghl)
#### Post date: [October 17, 2019, 8:03am UTC](https://forums.suse.com/t/stopping-pacemaker-doesnt-move-resources-to-other-node/34131/13 "2019-10-17T08:03:35Z")
Hi,
Finally, I could test with fence\_scsi and it worked to the extent it is supposed to. But in order to unfence the node it requires the node to be rebooted, which requires a manual attention (or can be automated in some ways), but thats not probably I am looking at.
My requirement is -
1. node should not go down with fencing
2. unfencing operation should not require a reboot of the node
1 could be done by fence\_scsi, but 2) I couldnt find a way.
So, if 1) and 2) both are not possible, then I am fine going for sbd based fencing, but in that case I am looking for fence operation to actually shutdown the node instead of rebooting, but rebooting of the node is actually orchestrated from some other flow.
Is there a way I can achieve this ?
---
### Author:  [@strahil-nikolov-dxc](https://forums.suse.com/u/strahil-nikolov-dxc)
#### Post date: [October 19, 2019, 8:04pm UTC](https://forums.suse.com/t/stopping-pacemaker-doesnt-move-resources-to-other-node/34131/14 "2019-10-19T20:04:58Z")
Can you show the fencing device configurration?
```auto
crm configure show
```
Most probably you donât have the âmeta provides=âunfencingââ to your fencing device.
---
### Author:  [@mnshsnghl](https://forums.suse.com/u/mnshsnghl)
#### Post date: [October 21, 2019, 8:46am UTC](https://forums.suse.com/t/stopping-pacemaker-doesnt-move-resources-to-other-node/34131/15 "2019-10-21T08:46:04Z")
[QUOTE=strahil-nikolov-dxc;58693]Can you show the fencing device configurration?
```auto
crm configure show
```
Most probably you donât have the âmeta provides=âunfencingââ to your fencing device.[/QUOTE]
I dont have the configuration wrt fence\_scsi right now, coz I started experimenting with something else. I will try this option, but I still dont know even if I configure this option what will trigger unfencing operation to this node ? I read at multiple places â âThe failed node will no longer be able to write to the device(s). A manual reboot is required.â
Since reboot was not an option for me, I didnt configure that. But doesnt this sentence mean that a manual reboot needs to be triggered to the node, and during the course of reboot (maybe be before rebooting), the node shall be unfenced.
What I would want is to be able to âunfenceâ the node without rebooting. Is there a way to do it ? Can a node unfence itself once the issue with the node is resolved ?
Regards
Maneesh
---
### Author:  [@strahil-nikolov-dxc](https://forums.suse.com/u/strahil-nikolov-dxc)
#### Post date: [October 30, 2019, 5:23am UTC](https://forums.suse.com/t/stopping-pacemaker-doesnt-move-resources-to-other-node/34131/16 "2019-10-30T05:23:19Z")
[QUOTE=mnshsnghl;58698]I dont have the configuration wrt fence\_scsi right now, coz I started experimenting with something else. I will try this option, but I still dont know even if I configure this option what will trigger unfencing operation to this node ? I read at multiple places â âThe failed node will no longer be able to write to the device(s). A manual reboot is required.â
Since reboot was not an option for me, I didnt configure that. But doesnt this sentence mean that a manual reboot needs to be triggered to the node, and during the course of reboot (maybe be before rebooting), the node shall be unfenced.
What I would want is to be able to âunfenceâ the node without rebooting. Is there a way to do it ? Can a node unfence itself once the issue with the node is resolved ?
Regards
Maneesh[/QUOTE]
Unfencing should be done by pacemaker itself with the meta attribute from the previous comments. Sadly my test cluster is ontop of VmWare and that doesnât support persistent SCSI-3 reservations.
If I have time, I will deploy an iSCSI and try it myself.
Keep in mind that the fence\_scsi will automatically detect which LUNs need to be fenced/unfenced if they are part of a volume group with âcâ flag (clustered) , but will require the âdevices=â if you use HA-LVM (which I prefer as it supports dual corosync rings).