Load order for services

I am so close to getting this finished. This will be a high availability setup with Sles 12 SP5 and HA on top. Will be using it to manage virtual machines. Will use sanlock for disk locking to keep the machines from booting on 2 machine at once. I have 1 snag left that trying to figure out. Sanlock is required. The issue becomes that I have to have sanlock start after pacemaker. Otherwise pacemaker does not start and no cluster and no vm's. I have googled this but still get confused. Yes coming from Sles 11 which personally I liked batter for Xen virtual environment with built in disk locking. Any recommendations would be greatly appreciated.

Comments

  • malcolmlewismalcolmlewis Knowledge Partner

    Hi
    Assuming your using systemd services, then you would ensure sanlock service has an After=pacemaker.target entry. If you take this route ensure you copy the service file to /etc/system/systemd directory so it doesn't get over written in an update.

  • warper2warper2 Member

    Call me a dummy. But I am confused on what you are saying to do.

  • malcolmlewismalcolmlewis Knowledge Partner

    Hi
    So how are the pacemaker and sanlock services started on boot?

  • warper2warper2 Member

    Yes currently the are set in services mgr though last to start on boot. But pacemaker is loading after sanlock so does not come up since sanlock needs to be after.

  • warper2warper2 Member

    Ok trying this.
    Created a folder /etc/systemd/system/sanlock.service.requires
    did a symlink in the folder "ln -s /usr/lib/systemd/system/pacemaker.service pacemaker.service"
    rebooting to see if that is what need.

  • warper2warper2 Member

    That did not work

  • malcolmlewismalcolmlewis Knowledge Partner

    Hi
    Not that way, can you show the files sanlock.service and pacemaker.service?

  • warper2warper2 Member
    edited July 24

    deleted this comment

  • warper2warper2 Member

    Not sure why some text went bold and larger

  • malcolmlewismalcolmlewis Knowledge Partner
    edited July 24

    Hi
    Can you edit and add Markdown three ` at beginning and three ` at the end of your code output?

  • warper2warper2 Member

    So Markdown at beginning of the file and Markdown three at the end of the file. Making sure I understand. Then reboot after?

  • warper2warper2 Member

    On both files?

  • malcolmlewismalcolmlewis Knowledge Partner

    Hi
    No I meant in your post that is affected by big fonts ;) If not able to, can your repost the service files.

  • warper2warper2 Member
    edited July 24

    Can't seem to attach those kind of files.

  • malcolmlewismalcolmlewis Knowledge Partner
    edited July 24

    Hi
    No it's three of these ` as in ```

  • warper2warper2 Member
    edited July 24
    [Unit]
    Description=Pacemaker High Availability Cluster Manager
    Documentation=man:pacemakerd
    Documentation=https://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/1.1/html-single/Pacemaker_Explained/index.html
    
    # DefaultDependencies takes care of sysinit.target,
    # basic.target, and shutdown.target
    
    # We need networking to bind to a network address. It is recommended not to
    # use Wants or Requires with network.target, and not to use
    # network-online.target for server daemons.
    After=network.target
    
    # Time syncs can make the clock jump backward, which messes with logging
    # and failure timestamps, so wait until it's done.
    After=time-sync.target
    
    # Managing systemd resources requires DBus.
    After=dbus.service
    Wants=dbus.service
    
    # Some OCF resources may have dependencies that aren't managed by the cluster;
    # these must be started before Pacemaker and stopped after it. The
    # resource-agents package provides this target, which lets system adminstrators
    # add drop-ins for those dependencies.
    After=resource-agents-deps.target
    Wants=resource-agents-deps.target
    
    After=syslog.service
    After=rsyslog.service
    After=corosync.service
    Requires=corosync.service
    
    
    [Install]
    WantedBy=multi-user.target
    
    
    [Service]
    Type=simple
    KillMode=process
    NotifyAccess=main
    EnvironmentFile=-/etc/sysconfig/pacemaker
    EnvironmentFile=-/etc/sysconfig/sbd
    SuccessExitStatus=100
    
    ExecStart=/usr/sbin/pacemakerd -f
    
    # Systemd v227 and above can limit the number of processes spawned by a
    # service. That is a bad idea for an HA cluster resource manager, so disable it
    # by default. The administrator can create a local override if they really want
    # a limit. If your systemd version does not support TasksMax, and you want to
    # get rid of the resulting log warnings, comment out this option.
    TasksMax=infinity
    
    # If pacemakerd doesn't stop, it's probably waiting on a cluster
    # resource.  Sending -KILL will just get the node fenced
    SendSIGKILL=no
    
    # If we ever hit the StartLimitInterval/StartLimitBurst limit and the
    # admin wants to stop the cluster while pacemakerd is not running, it
    # might be a good idea to enable the ExecStopPost directive below.
    #
    # Although the node will likely end up being fenced as a result so it's
    # not on by default
    #
    # ExecStopPost=/usr/bin/killall -TERM crmd attrd stonithd cib pengine lrmd
    
    # If you want Corosync to stop whenever Pacemaker is stopped,
    # uncomment the next line too:
    #
    # ExecStopPost=/bin/sh -c 'pidof crmd || killall -TERM corosync'
    
    # Uncomment this for older versions of systemd that didn't support
    # TimeoutStopSec
    # TimeoutSec=30min
    
    # Pacemaker can only exit after all managed services have shut down
    # A HA database could conceivably take even longer than this 
    TimeoutStopSec=30min
    TimeoutStartSec=60s
    
    # Restart options include: no, on-success, on-failure, on-abort or always
    Restart=on-failure
    
    # crm_perror() writes directly to stderr, so ignore it here
    # to avoid double-logging with the wrong format
    StandardError=null
    
  • warper2warper2 Member
    [Unit]
    Description=Shared Storage Lease Manager
    After=syslog.target pacemaker.service
    Wants=wdmd.service pacemaker.service
    
    [Service]
    Type=forking
    ControlGroup=cpu:/
    EnvironmentFile=-/etc/sysconfig/sanlock
    ExecStart=/usr/sbin/sanlock daemon $SANLOCKOPTS
    ExecStop=/bin/kill -TERM $MAINPID
    ExecReload=/bin/kill -HUP $MAINPID
    
    [Install]
    WantedBy=multi-user.target
    
  • warper2warper2 Member

    I think I understand now :)

  • malcolmlewismalcolmlewis Knowledge Partner

    Hi
    So in the sanlock systemd service, it does contain pacemaker.service, or is that what you added?
    Can you show the status of the services?

    systemctl status pacemaker sanlock
    
  • warper2warper2 Member

    Current sanlock.service file pacemaker has not been changed

    [Unit]
    Description=Shared Storage Lease Manager
    After=syslog.target 
    After=pacemaker.service
    Wants=wdmd.service
    Requires=pacemaker.service
    
    [Service]
    Type=forking
    ControlGroup=cpu:/
    EnvironmentFile=-/etc/sysconfig/sanlock
    ExecStart=/usr/sbin/sanlock daemon $SANLOCKOPTS
    ExecStop=/bin/kill -TERM $MAINPID
    ExecReload=/bin/kill -HUP $MAINPID
    
    [Install]
    WantedBy=multi-user.target
    

    This is with sanlock enabled in yast, system services enables for sanlock

    linuxvm2:~ # systemctl status pacemaker sanlock
    ● pacemaker.service - Pacemaker High Availability Cluster Manager
       Loaded: loaded (/usr/lib/systemd/system/pacemaker.service; enabled; vendor preset: disabled)
       Active: inactive (dead)
         Docs: man:pacemakerd
               https://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/1.1/html-single/Pacemaker_Explained/index.html
    
    Jul 24 12:18:38 linuxvm2 systemd[1]: Dependency failed for Pacemaker High Availability Cluster Manager.
    Jul 24 12:18:38 linuxvm2 systemd[1]: pacemaker.service: Job pacemaker.service/start failed with result 'dependency'.
    
    ● sanlock.service - Shared Storage Lease Manager
       Loaded: loaded (/usr/lib/systemd/system/sanlock.service; enabled; vendor preset: disabled)
       Active: inactive (dead)
    
    Jul 24 12:16:04 linuxvm2 systemd[1]: [/usr/lib/systemd/system/sanlock.service:10] Unknown lvalue 'ControlGroup' in section 'Service'
    Jul 24 12:18:38 linuxvm2 systemd[1]: Dependency failed for Shared Storage Lease Manager.
    Jul 24 12:18:38 linuxvm2 systemd[1]: sanlock.service: Job sanlock.service/start failed with result 'dependency'.
    linuxvm2:~ # 
    

    If I boot with sanlock disabled in system services pacemaker comes up fine and I can than start sanlock
    manually with systemctl start sanlock.service

    If I shutdown pacemaker and than try systemctl start sanlock.service it will not start.
    I am using sbd as well which of course effects everything loading as well.

  • malcolmlewismalcolmlewis Knowledge Partner

    Hi
    Looks like you need to fix this error first?
    Jul 24 12:16:04 linuxvm2 systemd[1]: [/usr/lib/systemd/system/sanlock.service:10] Unknown lvalue 'ControlGroup' in section 'Service'

  • warper2warper2 Member

    That is what is installed there by default. I have 2 servers. Only editing one till get this figured out. I would have no idea what it would be changed to.

  • warper2warper2 Member

    Ultimately this is what I see.
    system boots with sanlock disabled
    pacemaker loads which also loads sbd I believe which in turn loads wdmd.
    with sanlock enabled the system boots and I think sanlock is than coming up first which than causes wdmd to load which than cause sbd to not load and than pacemaker will not start.

  • warper2warper2 Member

    This is with sanlock disabled and I manually start after server comes up.

    linuxvm2:~ # rcpacemaker status
    * pacemaker.service - Pacemaker High Availability Cluster Manager
       Loaded: loaded (/usr/lib/systemd/system/pacemaker.service; enabled; vendor preset: disabled)
       Active: active (running) since Fri 2020-07-24 13:37:25 CDT; 2min 41s ago
         Docs: man:pacemakerd
               https://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/1.1/html-single/Pacemaker_Explained/index.html
     Main PID: 3004 (pacemakerd)
        Tasks: 7
       CGroup: /system.slice/pacemaker.service
               |-3004 /usr/sbin/pacemakerd -f
               |-3033 /usr/lib/pacemaker/cib
               |-3034 /usr/lib/pacemaker/stonithd
               |-3036 /usr/lib/pacemaker/lrmd
               |-3037 /usr/lib/pacemaker/attrd
               |-3038 /usr/lib/pacemaker/pengine
               `-3039 /usr/lib/pacemaker/crmd
    
    Jul 24 13:37:51 linuxvm2 crmd[3039]:   notice: Result of probe operation for Server9-start on linuxvm2: 7 (not running)
    Jul 24 13:37:51 linuxvm2 crmd[3039]:   notice: Initiating monitor operation webctrl-start_monitor_0 locally on linuxvm2
    Jul 24 13:37:51 linuxvm2 crmd[3039]:   notice: Result of probe operation for Starcall-start on linuxvm2: 7 (not running)
    Jul 24 13:37:51 linuxvm2 crmd[3039]:   notice: Initiating monitor operation windows2-start_monitor_0 locally on linuxvm2
    Jul 24 13:37:51 linuxvm2 crmd[3039]:   notice: Result of probe operation for webctrl-start on linuxvm2: 7 (not running)
    Jul 24 13:37:51 linuxvm2 crmd[3039]:   notice: Initiating monitor operation windows3-start_monitor_0 locally on linuxvm2
    Jul 24 13:37:52 linuxvm2 crmd[3039]:   notice: Result of probe operation for windows3-start on linuxvm2: 7 (not running)
    Jul 24 13:37:52 linuxvm2 crmd[3039]:   notice: Result of probe operation for windows2-start on linuxvm2: 7 (not running)
    Jul 24 13:37:52 linuxvm2 crmd[3039]:   notice: Transition 1 (Complete=24, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-warn-18.bz2): Complete
    Jul 24 13:37:52 linuxvm2 crmd[3039]:   notice: State transition S_TRANSITION_ENGINE -> S_IDLE
    linuxvm2:~ # rcsanlock status
    Checking status of sanlock                                                                                                                        unused
    ● sanlock.service - Shared Storage Lease Manager
       Loaded: loaded (/usr/lib/systemd/system/sanlock.service; disabled; vendor preset: disabled)
       Active: inactive (dead)
    linuxvm2:~ # rcsanlock start
    redirecting to systemctl start sanlock.service
    linuxvm2:~ # rcsanlock status
    Checking status of sanlock                                                                                                                        running
    ● sanlock.service - Shared Storage Lease Manager
       Loaded: loaded (/usr/lib/systemd/system/sanlock.service; disabled; vendor preset: disabled)
       Active: active (running) since Fri 2020-07-24 13:40:19 CDT; 8s ago
      Process: 4011 ExecStart=/usr/sbin/sanlock daemon $SANLOCKOPTS (code=exited, status=0/SUCCESS)
     Main PID: 4012 (sanlock)
        Tasks: 6 (limit: 512)
       CGroup: /system.slice/sanlock.service
               ├─4012 /usr/sbin/sanlock daemon -w 0
               └─4013 /usr/sbin/sanlock daemon -w 0
    linuxvm2:~ # 
    
  • malcolmlewismalcolmlewis Knowledge Partner

    Hi
    I would tempted to script the startup with a sleep to ensure pacemaker is started (or look for the pacemaker status) and use your own service file in /etc/systemd/system....

    Else, I would also suggest a support request to SUSE if you can, I wonder if it's because one is type simple and the other forking which maybe causing issues.

  • warper2warper2 Member

    After days and many hours of beating my head against the wall. I have finally figured it out. I used this website to try different things. I have learned a lot through this ordeal. I used the custom.target one in the site. Took a while to figure out though that sanlock seems to require a forking startup type. Thank you for the help to figure different searches to find what needed.
    https://superuser.com/questions/544399/how-do-you-make-a-systemd-service-as-the-last-service-on-boot

  • malcolmlewismalcolmlewis Knowledge Partner

    Hi
    Thanks for posting back your solution and your welcome glad to be of some help :)

  • warper2warper2 Member

    Just so aware that did not totally resolve the issue. It still could not talk to watchdog so was really flaky. Went back to lockd.

Sign In or Register to comment.