Results 1 to 2 of 2

Thread: HA Resource for CTDB fails after Upgrade to SP4

Hybrid View

  1. #1

    Question HA Resource for CTDB fails after Upgrade to SP4

    Hi folks,

    I have a 2-node HA Cluster for file services (samba, nfs). After Upgrading one of them so SLES12 SP4 (HA SP4), the CTDB-Resource doesn't start on the Upgraded host.
    It's because of the wrong ctdbd startup paramater "--logfile=/var/log/ctdb/log.ctdb" (instead of "--logging=file:/var/log/ctdb/log.ctdb")

    Any advice on solving this cluster issue?

    Thanks!

    Log from node with SP4:
    +++ 15:04:11: ocf_start_trace:1000: echo
    +++ 15:04:11: ocf_start_trace:1000: printenv
    +++ 15:04:11: ocf_start_trace:1000: sort
    ++ 15:04:11: ocf_start_trace:1000: env='
    HA_LOGD=no
    HA_LOGFACILITY=daemon
    HA_cluster_type=corosync
    HA_debug=0
    HA_logfacility=daemon
    HA_mcp=true
    HA_quorum_type=corosync
    HA_use_logd=off
    LC_ALL=C
    LC_CTYPE=de_DE.UTF-8
    LRMD_MAX_CHILDREN=4
    OCF_EXIT_REASON_PREFIX=ocf-exit-reason:
    OCF_RA_VERSION_MAJOR=1
    OCF_RA_VERSION_MINOR=0
    OCF_RESKEY_CRM_meta_clone=0
    OCF_RESKEY_CRM_meta_clone_max=2
    OCF_RESKEY_CRM_meta_clone_node_max=1
    OCF_RESKEY_CRM_meta_globally_unique=false
    OCF_RESKEY_CRM_meta_name=start
    OCF_RESKEY_CRM_meta_notify=false
    OCF_RESKEY_CRM_meta_on_node=fileserver2t
    OCF_RESKEY_CRM_meta_on_node_uuid=740214560
    OCF_RESKEY_CRM_meta_timeout=90000
    OCF_RESKEY_CRM_meta_trace_ra=1
    OCF_RESKEY_crm_feature_set=3.0.13
    OCF_RESKEY_ctdb_manages_samba=false
    OCF_RESKEY_ctdb_manages_winbind=false
    OCF_RESKEY_ctdb_recovery_lock=/data/ctdb/samba/ctdb.lock
    OCF_RESKEY_ctdb_socket=/var/lib/ctdb/ctdb.socket
    OCF_RESKEY_trace_ra=1
    OCF_RESOURCE_INSTANCE=res_CTDB
    OCF_RESOURCE_PROVIDER=heartbeat
    OCF_RESOURCE_TYPE=CTDB
    OCF_ROOT=/usr/lib/ocf
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/ucb
    PCMK_cluster_type=corosync
    PCMK_debug=0
    PCMK_logfacility=daemon
    PCMK_mcp=true
    PCMK_quorum_type=corosync
    PCMK_service=lrmd
    PCMK_use_logd=off
    PCMK_watchdog=true
    PWD=/var/lib/pacemaker/cores
    SBD_DELAY_START=
    SBD_DEVICE=/dev/mapper/360002ac000000000000002260001a15d-part1
    SBD_OPTS=
    SBD_PACEMAKER=
    SBD_STARTMODE=
    SBD_WATCHDOG=
    SHLVL=1
    _=/usr/bin/printenv
    __OCF_TRC_DEST=/var/lib/heartbeat/trace_ra/CTDB/res_CTDB.start.2019-04-15.15:04:11
    __OCF_TRC_MANAGE=1'
    ++ 15:04:11: source:1054: ocf_is_true off
    ++ 15:04:11: ocf_is_true:103: case "$1" in
    ++ 15:04:11: ocf_is_true:103: case "$1" in
    ++ 15:04:11: ocf_is_true:105: false
    + 15:04:11: main:78: var_prefix=/var/lib/ctdb
    + 15:04:11: main:79: '[' '!' -d /var/lib/ctdb ']'
    + 15:04:11: main:83: run_prefix=/run
    + 15:04:11: main:84: '[' '!' -d /var/lib/ctdb ']'
    + 15:04:11: main:88: : false
    + 15:04:11: main:89: : false
    + 15:04:11: main:90: :
    + 15:04:11: main:91: :
    + 15:04:11: main:92: :
    + 15:04:11: main:93: : yes
    + 15:04:11: main:94: : 100
    + 15:04:11: main:95: : no
    + 15:04:11: main:97: : /etc/ctdb
    + 15:04:11: main:98: : /usr/bin/ctdb
    + 15:04:11: main:99: : /usr/sbin/ctdbd
    + 15:04:11: main:100: : /var/lib/ctdb
    + 15:04:11: main:101: : /var/log/ctdb/log.ctdb
    + 15:04:11: main:102: : /run/ctdb
    + 15:04:11: main:103: : /var/lib/ctdb/ctdb.socket
    + 15:04:11: main:104: : 2
    + 15:04:11: main:106: : /etc/samba/smb.conf
    + 15:04:11: main:107: : tdbsam
    + 15:04:11: main:108: : tdb2
    + 15:04:11: main:358: '[' -f /etc/sysconfig/ctdb ']'
    + 15:04:11: main:359: CTDB_SYSCONFIG=/etc/sysconfig/ctdb
    + 15:04:11: main:369: CTDB_SYSCONFIG_BACKUP=/etc/sysconfig/ctdb.ctdb-ra-orig
    + 15:04:11: main:783: case $__OCF_ACTION in
    + 15:04:11: main:787: ctdb_start
    + 15:04:11: ctdb_start:561: ctdb_monitor
    + 15:04:11: ctdb_monitor:702: local status
    ++ 15:04:11: ctdb_monitor:708: invoke_ctdb status
    ++ 15:04:11: invoke_ctdb:373: local timeout
    ++ 15:04:11: invoke_ctdb:374: local timelimit
    ++ 15:04:11: invoke_ctdb:375: timeout=3
    ++ 15:04:11: invoke_ctdb:376: timelimit=120
    ++ 15:04:11: invoke_ctdb:378: '[' -n 90000 ']'
    ++ 15:04:11: invoke_ctdb:379: timeout=90
    ++ 15:04:11: invoke_ctdb:380: timelimit=90
    ++ 15:04:11: invoke_ctdb:382: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
    + 15:04:11: ctdb_monitor:708: status='connect() failed, errno=111
    Failed to connect to CTDB daemon (/var/lib/ctdb/ctdb.socket)'
    + 15:04:11: ctdb_monitor:709: '[' 1 -ne 0 ']'
    + 15:04:11: ctdb_monitor:710: echo 'connect() failed, errno=111
    Failed to connect to CTDB daemon (/var/lib/ctdb/ctdb.socket)'
    + 15:04:11: ctdb_monitor:710: grep -qs 'Connection refused'
    + 15:04:11: ctdb_monitor:712: echo 'connect() failed, errno=111
    Failed to connect to CTDB daemon (/var/lib/ctdb/ctdb.socket)'
    + 15:04:11: ctdb_monitor:712: grep -qs 'No such file or directory'
    + 15:04:11: ctdb_monitor:714: echo 'connect()' failed, errno=111 Failed to connect to CTDB daemon '(/var/lib/ctdb/ctdb.socket)'
    + 15:04:11: ctdb_monitor:714: grep -qs 'connect() failed'
    + 15:04:11: ctdb_monitor:715: return 7
    + 15:04:11: ctdb_start:564: ctdb_validate
    + 15:04:11: ctdb_validate:732: for binary in pkill
    + 15:04:11: ctdb_validate:733: check_binary pkill
    + 15:04:11: check_binary:55: have_binary pkill
    + 15:04:11: have_binary:67: '[' '' = 1 ']'
    ++ 15:04:11: have_binary:70: echo pkill
    ++ 15:04:11: have_binary:70: sed -e 's/ -.*//'
    + 15:04:11: have_binary:70: local bin=pkill
    ++ 15:04:11: have_binary:71: which pkill
    + 15:04:11: have_binary:71: test -x /usr/bin/pkill
    + 15:04:11: ctdb_validate:736: '[' -z /etc/sysconfig/ctdb ']'
    + 15:04:11: ctdb_validate:741: ocf_is_true false
    + 15:04:11: ocf_is_true:103: case "$1" in
    + 15:04:11: ocf_is_true:105: false
    + 15:04:11: ctdb_validate:746: '[' -f /etc/ctdb/public_addresses ']'
    + 15:04:11: ctdb_validate:750: '[' '!' -f /etc/ctdb/nodes ']'
    + 15:04:11: ctdb_validate:755: '[' -z /data/ctdb/samba/ctdb.lock ']'
    + 15:04:11: ctdb_validate:760: '[' / == '!' ']'
    ++ 15:04:11: ctdb_validate:770: dirname /data/ctdb/samba/ctdb.lock
    ++ 15:04:11: dirname:654: local a
    ++ 15:04:11: dirname:655: local b
    ++ 15:04:11: dirname:657: '[' 1 = 1 ']'
    ++ 15:04:11: dirname:658: a=/data/ctdb/samba/ctdb.lock
    ++ 15:04:11: dirname:659: '[' 1 ']'
    ++ 15:04:11: dirname:660: b=/data/ctdb/samba/ctdb.lock
    ++ 15:04:11: dirname:661: '[' /data/ctdb/samba/ctdb.lock = /data/ctdb/samba/ctdb.lock ']'
    ++ 15:04:11: dirname:661: break
    ++ 15:04:11: dirname:664: b=/data/ctdb/samba
    ++ 15:04:11: dirname:665: '[' -z /data/ctdb/samba -o /data/ctdb/samba/ctdb.lock = /data/ctdb/samba ']'
    ++ 15:04:11: dirname:667: echo /data/ctdb/samba
    ++ 15:04:11: dirname:668: return 0
    + 15:04:11: ctdb_validate:770: lock_dir=/data/ctdb/samba
    + 15:04:11: ctdb_validate:771: touch /data/ctdb/samba/11867
    + 15:04:11: ctdb_validate:772: '[' 0 '!=' 0 ']'
    + 15:04:11: ctdb_validate:776: rm /data/ctdb/samba/11867
    + 15:04:11: ctdb_validate:779: return 0
    + 15:04:11: ctdb_start:565: rv=0
    + 15:04:11: ctdb_start:566: '[' 0 -ne 0 ']'
    + 15:04:11: ctdb_start:569: init_smb_conf
    + 15:04:11: init_smb_conf:441: ocf_is_true false
    + 15:04:11: ocf_is_true:103: case "$1" in
    + 15:04:11: ocf_is_true:105: false
    + 15:04:11: init_smb_conf:441: return 0
    + 15:04:11: ctdb_start:570: '[' 0 -ne 0 ']'
    + 15:04:11: ctdb_start:576: generate_ctdb_sysconfig
    + 15:04:11: generate_ctdb_sysconfig:521: grep -qa '# CTDB-RA: Auto-generated' /etc/sysconfig/ctdb
    + 15:04:11: generate_ctdb_sysconfig:522: '[' 0 -ne 0 ']'
    + 15:04:11: generate_ctdb_sysconfig:526: ocf_log info 'Generating new /etc/sysconfig/ctdb'
    + 15:04:11: ocf_log:323: '[' 2 -lt 2 ']'
    + 15:04:11: ocf_log:327: __OCF_PRIO=info
    + 15:04:11: ocf_log:328: shift
    + 15:04:11: ocf_log:329: __OCF_MSG='Generating new /etc/sysconfig/ctdb'
    + 15:04:11: ocf_log:331: case "${__OCF_PRIO}" in
    + 15:04:11: ocf_log:335: __OCF_PRIO=INFO
    + 15:04:11: ocf_log:340: '[' INFO = DEBUG ']'
    + 15:04:11: ocf_log:343: ha_log 'INFO: Generating new /etc/sysconfig/ctdb'
    + 15:04:11: ha_log:255: __ha_log 'INFO: Generating new /etc/sysconfig/ctdb'
    + 15:04:11: __ha_log:187: local ignore_stderr=false
    + 15:04:11: __ha_log:188: local loglevel
    + 15:04:11: __ha_log:190: '[' 'xINFO: Generating new /etc/sysconfig/ctdb' = x--ignore-stderr ']'
    + 15:04:11: __ha_log:192: '[' none = daemon ']'
    + 15:04:11: __ha_log:194: tty
    + 15:04:11: __ha_log:209: set_logtag
    + 15:04:11: set_logtag:177: '[' -z '' ']'
    + 15:04:11: set_logtag:178: '[' -n res_CTDB ']'
    + 15:04:11: set_logtag:179: HA_LOGTAG='CTDB(res_CTDB)[11867]'
    + 15:04:11: __ha_log:211: '[' xno = xyes ']'
    + 15:04:11: __ha_log:219: '[' -n daemon ']'
    + 15:04:11: __ha_log:221: : logging through syslog
    + 15:04:11: __ha_log:223: loglevel=notice
    + 15:04:11: __ha_log:224: case "${*}" in
    + 15:04:11: __ha_log:227: loglevel=info
    + 15:04:11: __ha_log:229: logger -t 'CTDB(res_CTDB)[11867]' -p daemon.info 'INFO: Generating new /etc/sysconfig/ctdb'
    + 15:04:11: __ha_log:232: '[' -n '' ']'
    + 15:04:11: __ha_log:238: '[' -z daemon -a -z '' ']'
    + 15:04:11: __ha_log:244: '[' -n /dev/null ']'
    + 15:04:11: __ha_log:246: : appending to /dev/null
    + 15:04:11: __ha_log:247: '[' x '!=' /dev/nullx ']'
    ++ 15:04:11: __ha_log:248: hadate
    ++ 15:04:11: hadate:173: date '+%Y/%m/%d_%T '
    + 15:04:11: __ha_log:248: echo 'CTDB(res_CTDB)[11867]: 2019/04/15_15:04:11' 'INFO: Generating new /etc/sysconfig/ctdb'
    + 15:04:11: generate_ctdb_sysconfig:537: cat
    ++ 15:04:11: generate_ctdb_sysconfig:537: ocf_is_true yes
    ++ 15:04:11: ocf_is_true:103: case "$1" in
    ++ 15:04:11: ocf_is_true:103: case "$1" in
    ++ 15:04:11: ocf_is_true:104: true
    ++ 15:04:11: generate_ctdb_sysconfig:537: echo yes
    ++ 15:04:11: generate_ctdb_sysconfig:537: ocf_is_true false
    ++ 15:04:11: ocf_is_true:103: case "$1" in
    ++ 15:04:11: ocf_is_true:103: case "$1" in
    ++ 15:04:11: ocf_is_true:105: false
    ++ 15:04:11: generate_ctdb_sysconfig:537: echo no
    ++ 15:04:11: generate_ctdb_sysconfig:537: ocf_is_true false
    ++ 15:04:11: ocf_is_true:103: case "$1" in
    ++ 15:04:11: ocf_is_true:103: case "$1" in
    ++ 15:04:11: ocf_is_true:105: false
    ++ 15:04:11: generate_ctdb_sysconfig:537: echo no
    + 15:04:11: generate_ctdb_sysconfig:544: append_ctdb_sysconfig CTDB_SERVICE_SMB
    + 15:04:11: append_ctdb_sysconfig:514: '[' -n '' ']'
    + 15:04:11: generate_ctdb_sysconfig:545: append_ctdb_sysconfig CTDB_SERVICE_NMB
    + 15:04:11: append_ctdb_sysconfig:514: '[' -n '' ']'
    + 15:04:11: generate_ctdb_sysconfig:546: append_ctdb_sysconfig CTDB_SERVICE_WINBIND
    + 15:04:11: append_ctdb_sysconfig:514: '[' -n '' ']'
    + 15:04:11: ctdb_start:577: enable_event_scripts
    + 15:04:11: enable_event_scripts:392: local event_dir
    + 15:04:11: enable_event_scripts:393: event_dir=/etc/ctdb/events.d
    + 15:04:11: enable_event_scripts:395: chmod u+x /etc/ctdb/events.d/00.ctdb
    + 15:04:11: enable_event_scripts:397: '[' -f /etc/ctdb/public_addresses ']'
    + 15:04:11: enable_event_scripts:400: chmod a-x /etc/ctdb/events.d/10.interface
    + 15:04:12: enable_event_scripts:402: '[' -f /etc/ctdb/static-routes ']'
    + 15:04:12: enable_event_scripts:405: chmod a-x /etc/ctdb/events.d/11.routing
    + 15:04:12: enable_event_scripts:407: ocf_is_true false
    + 15:04:12: ocf_is_true:103: case "$1" in
    + 15:04:12: ocf_is_true:105: false
    + 15:04:12: enable_event_scripts:408: ocf_is_true false
    + 15:04:12: ocf_is_true:103: case "$1" in
    + 15:04:12: ocf_is_true:105: false
    + 15:04:12: enable_event_scripts:411: chmod a-x /etc/ctdb/events.d/50.samba
    + 15:04:12: ctdb_start:580: local log_option
    ++ 15:04:12: ctdb_start:583: ctdb version
    ++ 15:04:12: ctdb_start:583: awk '{print $NF}'
    ++ 15:04:12: ctdb_start:583: sed 's/\.\?[[:alpha:]].*//'
    + 15:04:12: ctdb_start:583: version=4.6.16-
    + 15:04:12: ctdb_start:584: ocf_version_cmp 4.6.16- 4.2.14
    + 15:04:12: ocf_version_cmp:606: ocf_is_ver 4.6.16-
    + 15:04:12: ocf_is_ver:576: echo 4.6.16-
    + 15:04:12: ocf_is_ver:576: grep '^[0-9][0-9.-]*[0-9]$'
    + 15:04:12: ocf_version_cmp:606: return 3
    + 15:04:12: ctdb_start:585: '[' 3 -eq 2 ']'
    + 15:04:12: ctdb_start:591: log_option=--logfile=/var/log/ctdb/log.ctdb
    + 15:04:12: ctdb_start:592: '[' /var/log/ctdb/log.ctdb = syslog ']'
    ++ 15:04:12: ctdb_start:596: dirname /var/log/ctdb/log.ctdb
    ++ 15:04:12: dirname:654: local a
    ++ 15:04:12: dirname:655: local b
    ++ 15:04:12: dirname:657: '[' 1 = 1 ']'
    ++ 15:04:12: dirname:658: a=/var/log/ctdb/log.ctdb
    ++ 15:04:12: dirname:659: '[' 1 ']'
    ++ 15:04:12: dirname:660: b=/var/log/ctdb/log.ctdb
    ++ 15:04:12: dirname:661: '[' /var/log/ctdb/log.ctdb = /var/log/ctdb/log.ctdb ']'
    ++ 15:04:12: dirname:661: break
    ++ 15:04:12: dirname:664: b=/var/log/ctdb
    ++ 15:04:12: dirname:665: '[' -z /var/log/ctdb -o /var/log/ctdb/log.ctdb = /var/log/ctdb ']'
    ++ 15:04:12: dirname:667: echo /var/log/ctdb
    ++ 15:04:12: dirname:668: return 0
    + 15:04:12: ctdb_start:596: '[' '!' -d /var/log/ctdb ']'
    + 15:04:12: ctdb_start:602: mkdir -p /run/ctdb
    + 15:04:12: ctdb_start:605: local pub_addr_option
    + 15:04:12: ctdb_start:606: pub_addr_option=
    + 15:04:12: ctdb_start:607: '[' -f /etc/ctdb/public_addresses ']'
    + 15:04:12: ctdb_start:610: local start_as_disabled
    + 15:04:12: ctdb_start:611: start_as_disabled=--start-as-disabled
    + 15:04:12: ctdb_start:612: ocf_is_true no
    + 15:04:12: ocf_is_true:103: case "$1" in
    + 15:04:12: ocf_is_true:105: false
    + 15:04:12: ctdb_start:612: start_as_disabled=
    + 15:04:12: ctdb_start:616: /usr/sbin/ctdbd --reclock=/data/ctdb/samba/ctdb.lock --nlist=/etc/ctdb/nodes --socket=/var/lib/ctdb/ctdb.socket --dbdir=/var/lib/ctdb --dbdir-persistent=/var/lib/ctdb/persistent --event-script-dir=/etc/ctdb/events.d --notification-script=/etc/ctdb/notify.sh --transport=tcp --logfile=/var/log/ctdb/log.ctdb -d 2
    + 15:04:12: ctdb_start:626: '[' 1 -ne 0 ']'
    + 15:04:12: ctdb_start:628: cleanup_smb_conf
    + 15:04:12: cleanup_smb_conf:505: ocf_is_true false
    + 15:04:12: ocf_is_true:103: case "$1" in
    + 15:04:12: ocf_is_true:105: false
    + 15:04:12: cleanup_smb_conf:505: return 0
    + 15:04:12: ctdb_start:630: ocf_exit_reason 'Failed to execute /usr/sbin/ctdbd.'
    + 15:04:12: ocf_exit_reason:356: local cookie=ocf-exit-reason:
    + 15:04:12: ocf_exit_reason:357: local fmt
    + 15:04:12: ocf_exit_reason:358: local msg
    + 15:04:12: ocf_exit_reason:365: case $# in
    + 15:04:12: ocf_exit_reason:367: fmt=%s
    + 15:04:12: ocf_exit_reason:377: '[' -z ocf-exit-reason: ']'
    ++ 15:04:12: ocf_exit_reason:382: printf %s 'Failed to execute /usr/sbin/ctdbd.'
    + 15:04:12: ocf_exit_reason:382: msg='Failed to execute /usr/sbin/ctdbd.'
    + 15:04:12: ocf_exit_reason:383: printf '%s%s\n' ocf-exit-reason: 'Failed to execute /usr/sbin/ctdbd.'
    + 15:04:12: ocf_exit_reason:384: __ha_log --ignore-stderr 'ERROR: Failed to execute /usr/sbin/ctdbd.'
    + 15:04:12: __ha_log:187: local ignore_stderr=false
    + 15:04:12: __ha_log:188: local loglevel
    + 15:04:12: __ha_log:190: '[' x--ignore-stderr = x--ignore-stderr ']'
    + 15:04:12: __ha_log:190: ignore_stderr=true
    + 15:04:12: __ha_log:190: shift
    + 15:04:12: __ha_log:192: '[' none = daemon ']'
    + 15:04:12: __ha_log:194: tty
    + 15:04:12: __ha_log:209: set_logtag
    + 15:04:12: set_logtag:177: '[' -z 'CTDB(res_CTDB)[11867]' ']'
    + 15:04:12: __ha_log:211: '[' xno = xyes ']'
    + 15:04:12: __ha_log:219: '[' -n daemon ']'
    + 15:04:12: __ha_log:221: : logging through syslog
    + 15:04:12: __ha_log:223: loglevel=notice
    + 15:04:12: __ha_log:224: case "${*}" in
    + 15:04:12: __ha_log:225: loglevel=err
    + 15:04:12: __ha_log:229: logger -t 'CTDB(res_CTDB)[11867]' -p daemon.err 'ERROR: Failed to execute /usr/sbin/ctdbd.'
    + 15:04:12: __ha_log:232: '[' -n '' ']'
    + 15:04:12: __ha_log:238: '[' -z daemon -a -z '' ']'
    + 15:04:12: __ha_log:244: '[' -n /dev/null ']'
    + 15:04:12: __ha_log:246: : appending to /dev/null
    + 15:04:12: __ha_log:247: '[' x '!=' /dev/nullx ']'
    ++ 15:04:12: __ha_log:248: hadate
    ++ 15:04:12: hadate:173: date '+%Y/%m/%d_%T '
    + 15:04:12: __ha_log:248: echo 'CTDB(res_CTDB)[11867]: 2019/04/15_15:04:12' 'ERROR: Failed to execute /usr/sbin/ctdbd.'
    + 15:04:12: ctdb_start:631: return 1
    + 15:04:12: main:798: rc=1
    + 15:04:12: main:799: ocf_log debug 'res_CTDB start : 1'
    + 15:04:12: ocf_log:323: '[' 2 -lt 2 ']'
    + 15:04:12: ocf_log:327: __OCF_PRIO=debug
    + 15:04:12: ocf_log:328: shift
    + 15:04:12: ocf_log:329: __OCF_MSG='res_CTDB start : 1'
    + 15:04:12: ocf_log:331: case "${__OCF_PRIO}" in
    + 15:04:12: ocf_log:336: __OCF_PRIO=DEBUG
    + 15:04:12: ocf_log:340: '[' DEBUG = DEBUG ']'
    + 15:04:12: ocf_log:341: ha_debug 'DEBUG: res_CTDB start : 1'
    + 15:04:12: ha_debug:260: '[' x0 = x0 ']'
    + 15:04:12: ha_debug:261: return 0
    + 15:04:12: main:800: exit 1




    Log from node with SP3:
    +++ 15:04:11: ocf_start_trace:933: echo
    +++ 15:04:11: ocf_start_trace:933: printenv
    +++ 15:04:11: ocf_start_trace:933: sort
    ++ 15:04:11: ocf_start_trace:933: env='
    HA_LOGD=no
    HA_LOGFACILITY=daemon
    HA_cluster_type=corosync
    HA_debug=0
    HA_logfacility=daemon
    HA_mcp=true
    HA_quorum_type=corosync
    HA_use_logd=off
    LC_ALL=C
    LC_CTYPE=de_DE.UTF-8
    LRMD_MAX_CHILDREN=4
    OCF_EXIT_REASON_PREFIX=ocf-exit-reason:
    OCF_RA_VERSION_MAJOR=1
    OCF_RA_VERSION_MINOR=0
    OCF_RESKEY_CRM_meta_clone=1
    OCF_RESKEY_CRM_meta_clone_max=2
    OCF_RESKEY_CRM_meta_clone_node_max=1
    OCF_RESKEY_CRM_meta_globally_unique=false
    OCF_RESKEY_CRM_meta_name=start
    OCF_RESKEY_CRM_meta_notify=false
    OCF_RESKEY_CRM_meta_on_node=fileserver1t
    OCF_RESKEY_CRM_meta_on_node_uuid=740214559
    OCF_RESKEY_CRM_meta_timeout=90000
    OCF_RESKEY_CRM_meta_trace_ra=1
    OCF_RESKEY_crm_feature_set=3.0.13
    OCF_RESKEY_ctdb_manages_samba=false
    OCF_RESKEY_ctdb_manages_winbind=false
    OCF_RESKEY_ctdb_recovery_lock=/data/ctdb/samba/ctdb.lock
    OCF_RESKEY_ctdb_socket=/var/lib/ctdb/ctdb.socket
    OCF_RESKEY_trace_ra=1
    OCF_RESOURCE_INSTANCE=res_CTDB
    OCF_RESOURCE_PROVIDER=heartbeat
    OCF_RESOURCE_TYPE=CTDB
    OCF_ROOT=/usr/lib/ocf
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/ucb
    PCMK_cluster_type=corosync
    PCMK_debug=0
    PCMK_logfacility=daemon
    PCMK_mcp=true
    PCMK_quorum_type=corosync
    PCMK_service=lrmd
    PCMK_use_logd=off
    PCMK_watchdog=true
    PWD=/var/lib/pacemaker/cores
    SBD_DELAY_START=
    SBD_DEVICE=/dev/mapper/360002ac000000000000002260001a15d-part1
    SBD_OPTS=
    SBD_PACEMAKER=
    SBD_STARTMODE=
    SBD_WATCHDOG=
    SHLVL=1
    _=/usr/bin/printenv
    __OCF_TRC_DEST=/var/lib/heartbeat/trace_ra/CTDB/res_CTDB.start.2019-04-15.15:04:11
    __OCF_TRC_MANAGE=1'
    ++ 15:04:11: source:945: ocf_is_true off
    ++ 15:04:11: ocf_is_true:102: case "$1" in
    ++ 15:04:11: ocf_is_true:102: case "$1" in
    ++ 15:04:11: ocf_is_true:104: false
    + 15:04:11: main:78: var_prefix=/var/lib/ctdb
    + 15:04:11: main:79: '[' '!' -d /var/lib/ctdb ']'
    + 15:04:11: main:83: run_prefix=/run
    + 15:04:11: main:84: '[' '!' -d /var/lib/ctdb ']'
    + 15:04:11: main:88: : false
    + 15:04:11: main:89: : false
    + 15:04:11: main:90: :
    + 15:04:11: main:91: :
    + 15:04:11: main:92: :
    + 15:04:11: main:93: : yes
    + 15:04:11: main:94: : 100
    + 15:04:11: main:95: : no
    + 15:04:11: main:97: : /etc/ctdb
    + 15:04:11: main:98: : /usr/bin/ctdb
    + 15:04:11: main:99: : /usr/sbin/ctdbd
    + 15:04:11: main:100: : /var/lib/ctdb
    + 15:04:11: main:101: : /var/log/ctdb/log.ctdb
    + 15:04:11: main:102: : /run/ctdb
    + 15:04:11: main:103: : /var/lib/ctdb/ctdb.socket
    + 15:04:11: main:104: : 2
    + 15:04:11: main:106: : /etc/samba/smb.conf
    + 15:04:11: main:107: : tdbsam
    + 15:04:11: main:108: : tdb2
    + 15:04:11: main:358: '[' -f /etc/sysconfig/ctdb ']'
    + 15:04:11: main:359: CTDB_SYSCONFIG=/etc/sysconfig/ctdb
    + 15:04:11: main:367: CTDB_SYSCONFIG_BACKUP=/etc/sysconfig/ctdb.ctdb-ra-orig
    + 15:04:11: main:757: case $__OCF_ACTION in
    + 15:04:11: main:761: ctdb_start
    + 15:04:11: ctdb_start:547: ctdb_monitor
    + 15:04:11: ctdb_monitor:676: local status
    ++ 15:04:11: ctdb_monitor:682: invoke_ctdb status
    ++ 15:04:11: invoke_ctdb:371: local timeout
    ++ 15:04:11: invoke_ctdb:372: local timelimit
    ++ 15:04:11: invoke_ctdb:373: timeout=3
    ++ 15:04:11: invoke_ctdb:374: timelimit=120
    ++ 15:04:11: invoke_ctdb:376: '[' -n 90000 ']'
    ++ 15:04:11: invoke_ctdb:377: timeout=90
    ++ 15:04:11: invoke_ctdb:378: timelimit=90
    ++ 15:04:11: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
    + 15:04:11: ctdb_monitor:682: status='connect() failed, errno=111
    Failed to connect to CTDB daemon (/var/lib/ctdb/ctdb.socket)'
    + 15:04:11: ctdb_monitor:683: '[' 1 -ne 0 ']'
    + 15:04:11: ctdb_monitor:684: echo 'connect() failed, errno=111
    Failed to connect to CTDB daemon (/var/lib/ctdb/ctdb.socket)'
    + 15:04:11: ctdb_monitor:684: grep -qs 'Connection refused'
    + 15:04:11: ctdb_monitor:686: echo 'connect() failed, errno=111
    Failed to connect to CTDB daemon (/var/lib/ctdb/ctdb.socket)'
    + 15:04:11: ctdb_monitor:686: grep -qs 'No such file or directory'
    + 15:04:11: ctdb_monitor:688: echo 'connect()' failed, errno=111 Failed to connect to CTDB daemon '(/var/lib/ctdb/ctdb.socket)'
    + 15:04:11: ctdb_monitor:688: grep -qs 'connect() failed'
    + 15:04:11: ctdb_monitor:689: return 7
    + 15:04:11: ctdb_start:550: ctdb_validate
    + 15:04:11: ctdb_validate:706: for binary in pkill
    + 15:04:11: ctdb_validate:707: check_binary pkill
    + 15:04:11: check_binary:56: have_binary pkill
    + 15:04:11: have_binary:68: '[' '' = 1 ']'
    ++ 15:04:11: have_binary:71: echo pkill
    ++ 15:04:11: have_binary:71: sed -e 's/ -.*//'
    + 15:04:11: have_binary:71: local bin=pkill
    ++ 15:04:11: have_binary:72: which pkill
    + 15:04:11: have_binary:72: test -x /usr/bin/pkill
    + 15:04:11: ctdb_validate:710: '[' -z /etc/sysconfig/ctdb ']'
    + 15:04:11: ctdb_validate:715: ocf_is_true false
    + 15:04:11: ocf_is_true:102: case "$1" in
    + 15:04:11: ocf_is_true:104: false
    + 15:04:11: ctdb_validate:720: '[' -f /etc/ctdb/public_addresses ']'
    + 15:04:11: ctdb_validate:724: '[' '!' -f /etc/ctdb/nodes ']'
    + 15:04:11: ctdb_validate:729: '[' -z /data/ctdb/samba/ctdb.lock ']'
    + 15:04:11: ctdb_validate:734: '[' / == '!' ']'
    ++ 15:04:11: ctdb_validate:744: dirname /data/ctdb/samba/ctdb.lock
    ++ 15:04:11: dirname:592: local a
    ++ 15:04:11: dirname:593: local b
    ++ 15:04:11: dirname:595: '[' 1 = 1 ']'
    ++ 15:04:11: dirname:596: a=/data/ctdb/samba/ctdb.lock
    ++ 15:04:11: dirname:597: '[' 1 ']'
    ++ 15:04:11: dirname:598: b=/data/ctdb/samba/ctdb.lock
    ++ 15:04:11: dirname:599: '[' /data/ctdb/samba/ctdb.lock = /data/ctdb/samba/ctdb.lock ']'
    ++ 15:04:11: dirname:599: break
    ++ 15:04:11: dirname:602: b=/data/ctdb/samba
    ++ 15:04:11: dirname:603: '[' -z /data/ctdb/samba -o /data/ctdb/samba/ctdb.lock = /data/ctdb/samba ']'
    ++ 15:04:11: dirname:605: echo /data/ctdb/samba
    ++ 15:04:11: dirname:606: return 0
    + 15:04:11: ctdb_validate:744: lock_dir=/data/ctdb/samba
    + 15:04:11: ctdb_validate:745: touch /data/ctdb/samba/14280
    + 15:04:11: ctdb_validate:746: '[' 0 '!=' 0 ']'
    + 15:04:11: ctdb_validate:750: rm /data/ctdb/samba/14280
    + 15:04:11: ctdb_validate:753: return 0
    + 15:04:11: ctdb_start:551: rv=0
    + 15:04:11: ctdb_start:552: '[' 0 -ne 0 ']'
    + 15:04:11: ctdb_start:555: init_smb_conf
    + 15:04:11: init_smb_conf:439: ocf_is_true false
    + 15:04:11: ocf_is_true:102: case "$1" in
    + 15:04:11: ocf_is_true:104: false
    + 15:04:11: init_smb_conf:439: return 0
    + 15:04:11: ctdb_start:556: '[' 0 -ne 0 ']'
    + 15:04:11: ctdb_start:562: generate_ctdb_sysconfig
    + 15:04:11: generate_ctdb_sysconfig:507: grep -qa '# CTDB-RA: Auto-generated' /etc/sysconfig/ctdb
    + 15:04:11: generate_ctdb_sysconfig:508: '[' 0 -ne 0 ']'
    + 15:04:11: generate_ctdb_sysconfig:512: ocf_log info 'Generating new /etc/sysconfig/ctdb'
    + 15:04:11: ocf_log:322: '[' 2 -lt 2 ']'
    + 15:04:11: ocf_log:326: __OCF_PRIO=info
    + 15:04:11: ocf_log:327: shift
    + 15:04:11: ocf_log:328: __OCF_MSG='Generating new /etc/sysconfig/ctdb'
    + 15:04:11: ocf_log:330: case "${__OCF_PRIO}" in
    + 15:04:11: ocf_log:334: __OCF_PRIO=INFO
    + 15:04:11: ocf_log:339: '[' INFO = DEBUG ']'
    + 15:04:11: ocf_log:342: ha_log 'INFO: Generating new /etc/sysconfig/ctdb'
    + 15:04:11: ha_log:254: __ha_log 'INFO: Generating new /etc/sysconfig/ctdb'
    + 15:04:11: __ha_log:186: local ignore_stderr=false
    + 15:04:11: __ha_log:187: local loglevel
    + 15:04:11: __ha_log:189: '[' 'xINFO: Generating new /etc/sysconfig/ctdb' = x--ignore-stderr ']'
    + 15:04:11: __ha_log:191: '[' none = daemon ']'
    + 15:04:11: __ha_log:193: tty
    + 15:04:11: __ha_log:208: set_logtag
    + 15:04:11: set_logtag:176: '[' -z '' ']'
    + 15:04:11: set_logtag:177: '[' -n res_CTDB ']'
    + 15:04:11: set_logtag:178: HA_LOGTAG='CTDB(res_CTDB)[14280]'
    + 15:04:11: __ha_log:210: '[' xno = xyes ']'
    + 15:04:11: __ha_log:218: '[' -n daemon ']'
    + 15:04:11: __ha_log:220: : logging through syslog
    + 15:04:11: __ha_log:222: loglevel=notice
    + 15:04:11: __ha_log:223: case "${*}" in
    + 15:04:11: __ha_log:226: loglevel=info
    + 15:04:11: __ha_log:228: logger -t 'CTDB(res_CTDB)[14280]' -p daemon.info 'INFO: Generating new /etc/sysconfig/ctdb'
    + 15:04:12: __ha_log:231: '[' -n '' ']'
    + 15:04:12: __ha_log:237: '[' -z daemon -a -z '' ']'
    + 15:04:12: __ha_log:243: '[' -n /dev/null ']'
    + 15:04:12: __ha_log:245: : appending to /dev/null
    + 15:04:12: __ha_log:246: '[' x '!=' /dev/nullx ']'
    ++ 15:04:12: __ha_log:247: hadate
    ++ 15:04:12: hadate:172: date '+%Y/%m/%d_%T '
    + 15:04:12: __ha_log:247: echo 'CTDB(res_CTDB)[14280]: 2019/04/15_15:04:12' 'INFO: Generating new /etc/sysconfig/ctdb'
    + 15:04:12: generate_ctdb_sysconfig:523: cat
    ++ 15:04:12: generate_ctdb_sysconfig:523: ocf_is_true yes
    ++ 15:04:12: ocf_is_true:102: case "$1" in
    ++ 15:04:12: ocf_is_true:102: case "$1" in
    ++ 15:04:12: ocf_is_true:103: true
    ++ 15:04:12: generate_ctdb_sysconfig:523: echo yes
    ++ 15:04:12: generate_ctdb_sysconfig:523: ocf_is_true false
    ++ 15:04:12: ocf_is_true:102: case "$1" in
    ++ 15:04:12: ocf_is_true:102: case "$1" in
    ++ 15:04:12: ocf_is_true:104: false
    ++ 15:04:12: generate_ctdb_sysconfig:523: echo no
    ++ 15:04:12: generate_ctdb_sysconfig:523: ocf_is_true false
    ++ 15:04:12: ocf_is_true:102: case "$1" in
    ++ 15:04:12: ocf_is_true:102: case "$1" in
    ++ 15:04:12: ocf_is_true:104: false
    ++ 15:04:12: generate_ctdb_sysconfig:523: echo no
    + 15:04:12: generate_ctdb_sysconfig:530: append_ctdb_sysconfig CTDB_SERVICE_SMB
    + 15:04:12: append_ctdb_sysconfig:500: '[' -n '' ']'
    + 15:04:12: generate_ctdb_sysconfig:531: append_ctdb_sysconfig CTDB_SERVICE_NMB
    + 15:04:12: append_ctdb_sysconfig:500: '[' -n '' ']'
    + 15:04:12: generate_ctdb_sysconfig:532: append_ctdb_sysconfig CTDB_SERVICE_WINBIND
    + 15:04:12: append_ctdb_sysconfig:500: '[' -n '' ']'
    + 15:04:12: ctdb_start:563: enable_event_scripts
    + 15:04:12: enable_event_scripts:390: local event_dir
    + 15:04:12: enable_event_scripts:391: event_dir=/etc/ctdb/events.d
    + 15:04:12: enable_event_scripts:393: chmod u+x /etc/ctdb/events.d/00.ctdb
    + 15:04:12: enable_event_scripts:395: '[' -f /etc/ctdb/public_addresses ']'
    + 15:04:12: enable_event_scripts:398: chmod a-x /etc/ctdb/events.d/10.interface
    + 15:04:12: enable_event_scripts:400: '[' -f /etc/ctdb/static-routes ']'
    + 15:04:12: enable_event_scripts:403: chmod a-x /etc/ctdb/events.d/11.routing
    + 15:04:12: enable_event_scripts:405: ocf_is_true false
    + 15:04:12: ocf_is_true:102: case "$1" in
    + 15:04:12: ocf_is_true:104: false
    + 15:04:12: enable_event_scripts:406: ocf_is_true false
    + 15:04:12: ocf_is_true:102: case "$1" in
    + 15:04:12: ocf_is_true:104: false
    + 15:04:12: enable_event_scripts:409: chmod a-x /etc/ctdb/events.d/50.samba
    + 15:04:12: ctdb_start:566: local log_option
    + 15:04:12: ctdb_start:567: log_option=--logging=file:/var/log/ctdb/log.ctdb
    + 15:04:12: ctdb_start:568: '[' /var/log/ctdb/log.ctdb = syslog ']'
    ++ 15:04:12: ctdb_start:570: dirname /var/log/ctdb/log.ctdb
    ++ 15:04:12: dirname:592: local a
    ++ 15:04:12: dirname:593: local b
    ++ 15:04:12: dirname:595: '[' 1 = 1 ']'
    ++ 15:04:12: dirname:596: a=/var/log/ctdb/log.ctdb
    ++ 15:04:12: dirname:597: '[' 1 ']'
    ++ 15:04:12: dirname:598: b=/var/log/ctdb/log.ctdb
    ++ 15:04:12: dirname:599: '[' /var/log/ctdb/log.ctdb = /var/log/ctdb/log.ctdb ']'
    ++ 15:04:12: dirname:599: break
    ++ 15:04:12: dirname:602: b=/var/log/ctdb
    ++ 15:04:12: dirname:603: '[' -z /var/log/ctdb -o /var/log/ctdb/log.ctdb = /var/log/ctdb ']'
    ++ 15:04:12: dirname:605: echo /var/log/ctdb
    ++ 15:04:12: dirname:606: return 0
    + 15:04:12: ctdb_start:570: '[' '!' -d /var/log/ctdb ']'
    + 15:04:12: ctdb_start:576: mkdir -p /run/ctdb
    + 15:04:12: ctdb_start:579: local pub_addr_option
    + 15:04:12: ctdb_start:580: pub_addr_option=
    + 15:04:12: ctdb_start:581: '[' -f /etc/ctdb/public_addresses ']'
    + 15:04:12: ctdb_start:584: local start_as_disabled
    + 15:04:12: ctdb_start:585: start_as_disabled=--start-as-disabled
    + 15:04:12: ctdb_start:586: ocf_is_true no
    + 15:04:12: ocf_is_true:102: case "$1" in
    + 15:04:12: ocf_is_true:104: false
    + 15:04:12: ctdb_start:586: start_as_disabled=
    + 15:04:12: ctdb_start:590: /usr/sbin/ctdbd --reclock=/data/ctdb/samba/ctdb.lock --nlist=/etc/ctdb/nodes --socket=/var/lib/ctdb/ctdb.socket --dbdir=/var/lib/ctdb --dbdir-persistent=/var/lib/ctdb/persistent --event-script-dir=/etc/ctdb/events.d --notification-script=/etc/ctdb/notify.sh --transport=tcp --logging=file:/var/log/ctdb/log.ctdb -d 2
    + 15:04:12: ctdb_start:600: '[' 0 -ne 0 ']'
    + 15:04:12: ctdb_start:609: true
    + 15:04:12: ctdb_start:613: sleep 1
    ++ 15:04:13: ctdb_start:614: invoke_ctdb status
    ++ 15:04:13: invoke_ctdb:371: local timeout
    ++ 15:04:13: invoke_ctdb:372: local timelimit
    ++ 15:04:13: invoke_ctdb:373: timeout=3
    ++ 15:04:13: invoke_ctdb:374: timelimit=120
    ++ 15:04:13: invoke_ctdb:376: '[' -n 90000 ']'
    ++ 15:04:13: invoke_ctdb:377: timeout=90
    ++ 15:04:13: invoke_ctdb:378: timelimit=90
    ++ 15:04:13: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
    + 15:04:13: ctdb_start:614: status='Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:INVALID
    Size:2
    hash:0 lmaster:0
    hash:1 lmaster:1
    Recovery mode:RECOVERY (1)
    Recovery master:-1'
    + 15:04:13: ctdb_start:615: '[' 0 -ne 0 ']'
    + 15:04:13: ctdb_start:621: echo 'Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:INVALID
    Size:2
    hash:0 lmaster:0
    hash:1 lmaster:1
    Recovery mode:RECOVERY (1)
    Recovery master:-1'
    + 15:04:13: ctdb_start:621: grep -qs 'UNHEALTHY (THIS'
    + 15:04:13: ctdb_start:609: true
    + 15:04:13: ctdb_start:613: sleep 1
    ++ 15:04:14: ctdb_start:614: invoke_ctdb status
    ++ 15:04:14: invoke_ctdb:371: local timeout
    ++ 15:04:14: invoke_ctdb:372: local timelimit
    ++ 15:04:14: invoke_ctdb:373: timeout=3
    ++ 15:04:14: invoke_ctdb:374: timelimit=120
    ++ 15:04:14: invoke_ctdb:376: '[' -n 90000 ']'
    ++ 15:04:14: invoke_ctdb:377: timeout=90
    ++ 15:04:14: invoke_ctdb:378: timelimit=90
    ++ 15:04:14: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
    + 15:04:14: ctdb_start:614: status='Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:INVALID
    Size:2
    hash:0 lmaster:0
    hash:1 lmaster:1
    Recovery mode:RECOVERY (1)
    Recovery master:0'
    + 15:04:14: ctdb_start:615: '[' 0 -ne 0 ']'
    + 15:04:14: ctdb_start:621: echo 'Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:INVALID
    Size:2
    hash:0 lmaster:0
    hash:1 lmaster:1
    Recovery mode:RECOVERY (1)
    Recovery master:0'
    + 15:04:14: ctdb_start:621: grep -qs 'UNHEALTHY (THIS'
    + 15:04:14: ctdb_start:609: true
    + 15:04:14: ctdb_start:613: sleep 1
    ++ 15:04:15: ctdb_start:614: invoke_ctdb status
    ++ 15:04:15: invoke_ctdb:371: local timeout
    ++ 15:04:15: invoke_ctdb:372: local timelimit
    ++ 15:04:15: invoke_ctdb:373: timeout=3
    ++ 15:04:15: invoke_ctdb:374: timelimit=120
    ++ 15:04:15: invoke_ctdb:376: '[' -n 90000 ']'
    ++ 15:04:15: invoke_ctdb:377: timeout=90
    ++ 15:04:15: invoke_ctdb:378: timelimit=90
    ++ 15:04:15: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
    + 15:04:15: ctdb_start:614: status='Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:INVALID
    Size:2
    hash:0 lmaster:0
    hash:1 lmaster:1
    Recovery mode:RECOVERY (1)
    Recovery master:0'
    + 15:04:15: ctdb_start:615: '[' 0 -ne 0 ']'
    + 15:04:15: ctdb_start:621: echo 'Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:INVALID
    Size:2
    hash:0 lmaster:0
    hash:1 lmaster:1
    Recovery mode:RECOVERY (1)
    Recovery master:0'
    + 15:04:15: ctdb_start:621: grep -qs 'UNHEALTHY (THIS'
    + 15:04:15: ctdb_start:609: true
    + 15:04:15: ctdb_start:613: sleep 1
    ++ 15:04:16: ctdb_start:614: invoke_ctdb status
    ++ 15:04:16: invoke_ctdb:371: local timeout
    ++ 15:04:16: invoke_ctdb:372: local timelimit
    ++ 15:04:16: invoke_ctdb:373: timeout=3
    ++ 15:04:16: invoke_ctdb:374: timelimit=120
    ++ 15:04:16: invoke_ctdb:376: '[' -n 90000 ']'
    ++ 15:04:16: invoke_ctdb:377: timeout=90
    ++ 15:04:16: invoke_ctdb:378: timelimit=90
    ++ 15:04:16: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
    + 15:04:16: ctdb_start:614: status='Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:INVALID
    Size:2
    hash:0 lmaster:0
    hash:1 lmaster:1
    Recovery mode:RECOVERY (1)
    Recovery master:0'
    + 15:04:16: ctdb_start:615: '[' 0 -ne 0 ']'
    + 15:04:16: ctdb_start:621: echo 'Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:INVALID
    Size:2
    hash:0 lmaster:0
    hash:1 lmaster:1
    Recovery mode:RECOVERY (1)
    Recovery master:0'
    + 15:04:16: ctdb_start:621: grep -qs 'UNHEALTHY (THIS'
    + 15:04:16: ctdb_start:609: true
    + 15:04:16: ctdb_start:613: sleep 1
    ++ 15:04:17: ctdb_start:614: invoke_ctdb status
    ++ 15:04:17: invoke_ctdb:371: local timeout
    ++ 15:04:17: invoke_ctdb:372: local timelimit
    ++ 15:04:17: invoke_ctdb:373: timeout=3
    ++ 15:04:17: invoke_ctdb:374: timelimit=120
    ++ 15:04:17: invoke_ctdb:376: '[' -n 90000 ']'
    ++ 15:04:17: invoke_ctdb:377: timeout=90
    ++ 15:04:17: invoke_ctdb:378: timelimit=90
    ++ 15:04:17: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
    + 15:04:17: ctdb_start:614: status='Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0'
    + 15:04:17: ctdb_start:615: '[' 0 -ne 0 ']'
    + 15:04:17: ctdb_start:621: echo 'Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0'
    + 15:04:17: ctdb_start:621: grep -qs 'UNHEALTHY (THIS'
    + 15:04:17: ctdb_start:609: true
    + 15:04:17: ctdb_start:613: sleep 1
    ++ 15:04:18: ctdb_start:614: invoke_ctdb status
    ++ 15:04:18: invoke_ctdb:371: local timeout
    ++ 15:04:18: invoke_ctdb:372: local timelimit
    ++ 15:04:18: invoke_ctdb:373: timeout=3
    ++ 15:04:18: invoke_ctdb:374: timelimit=120
    ++ 15:04:18: invoke_ctdb:376: '[' -n 90000 ']'
    ++ 15:04:18: invoke_ctdb:377: timeout=90
    ++ 15:04:18: invoke_ctdb:378: timelimit=90
    ++ 15:04:18: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
    + 15:04:18: ctdb_start:614: status='Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0'
    + 15:04:18: ctdb_start:615: '[' 0 -ne 0 ']'
    + 15:04:18: ctdb_start:621: echo 'Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0'
    + 15:04:18: ctdb_start:621: grep -qs 'UNHEALTHY (THIS'
    + 15:04:19: ctdb_start:609: true
    + 15:04:19: ctdb_start:613: sleep 1
    ++ 15:04:20: ctdb_start:614: invoke_ctdb status
    ++ 15:04:20: invoke_ctdb:371: local timeout
    ++ 15:04:20: invoke_ctdb:372: local timelimit
    ++ 15:04:20: invoke_ctdb:373: timeout=3
    ++ 15:04:20: invoke_ctdb:374: timelimit=120
    ++ 15:04:20: invoke_ctdb:376: '[' -n 90000 ']'
    ++ 15:04:20: invoke_ctdb:377: timeout=90
    ++ 15:04:20: invoke_ctdb:378: timelimit=90
    ++ 15:04:20: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
    + 15:04:20: ctdb_start:614: status='Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0'
    + 15:04:20: ctdb_start:615: '[' 0 -ne 0 ']'
    + 15:04:20: ctdb_start:621: echo 'Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0'
    + 15:04:20: ctdb_start:621: grep -qs 'UNHEALTHY (THIS'
    + 15:04:20: ctdb_start:609: true
    + 15:04:20: ctdb_start:613: sleep 1
    ++ 15:04:21: ctdb_start:614: invoke_ctdb status
    ++ 15:04:21: invoke_ctdb:371: local timeout
    ++ 15:04:21: invoke_ctdb:372: local timelimit
    ++ 15:04:21: invoke_ctdb:373: timeout=3
    ++ 15:04:21: invoke_ctdb:374: timelimit=120
    ++ 15:04:21: invoke_ctdb:376: '[' -n 90000 ']'
    ++ 15:04:21: invoke_ctdb:377: timeout=90
    ++ 15:04:21: invoke_ctdb:378: timelimit=90
    ++ 15:04:21: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
    + 15:04:21: ctdb_start:614: status='Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0'
    + 15:04:21: ctdb_start:615: '[' 0 -ne 0 ']'
    + 15:04:21: ctdb_start:621: echo 'Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0'
    + 15:04:21: ctdb_start:621: grep -qs 'UNHEALTHY (THIS'
    + 15:04:21: ctdb_start:609: true
    + 15:04:21: ctdb_start:613: sleep 1
    ++ 15:04:22: ctdb_start:614: invoke_ctdb status
    ++ 15:04:22: invoke_ctdb:371: local timeout
    ++ 15:04:22: invoke_ctdb:372: local timelimit
    ++ 15:04:22: invoke_ctdb:373: timeout=3
    ++ 15:04:22: invoke_ctdb:374: timelimit=120
    ++ 15:04:22: invoke_ctdb:376: '[' -n 90000 ']'
    ++ 15:04:22: invoke_ctdb:377: timeout=90
    ++ 15:04:22: invoke_ctdb:378: timelimit=90
    ++ 15:04:22: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
    + 15:04:22: ctdb_start:614: status='Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0'
    + 15:04:22: ctdb_start:615: '[' 0 -ne 0 ']'
    + 15:04:22: ctdb_start:621: echo 'Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0'
    + 15:04:22: ctdb_start:621: grep -qs 'UNHEALTHY (THIS'
    + 15:04:22: ctdb_start:609: true
    + 15:04:22: ctdb_start:613: sleep 1
    ++ 15:04:23: ctdb_start:614: invoke_ctdb status
    ++ 15:04:23: invoke_ctdb:371: local timeout
    ++ 15:04:23: invoke_ctdb:372: local timelimit
    ++ 15:04:23: invoke_ctdb:373: timeout=3
    ++ 15:04:23: invoke_ctdb:374: timelimit=120
    ++ 15:04:23: invoke_ctdb:376: '[' -n 90000 ']'
    ++ 15:04:23: invoke_ctdb:377: timeout=90
    ++ 15:04:23: invoke_ctdb:378: timelimit=90
    ++ 15:04:23: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
    + 15:04:23: ctdb_start:614: status='Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0'
    + 15:04:23: ctdb_start:615: '[' 0 -ne 0 ']'
    + 15:04:23: ctdb_start:621: echo 'Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0'
    + 15:04:23: ctdb_start:621: grep -qs 'UNHEALTHY (THIS'
    + 15:04:23: ctdb_start:609: true
    + 15:04:23: ctdb_start:613: sleep 1
    ++ 15:04:24: ctdb_start:614: invoke_ctdb status
    ++ 15:04:24: invoke_ctdb:371: local timeout
    ++ 15:04:24: invoke_ctdb:372: local timelimit
    ++ 15:04:24: invoke_ctdb:373: timeout=3
    ++ 15:04:24: invoke_ctdb:374: timelimit=120
    ++ 15:04:24: invoke_ctdb:376: '[' -n 90000 ']'
    ++ 15:04:24: invoke_ctdb:377: timeout=90
    ++ 15:04:24: invoke_ctdb:378: timelimit=90
    ++ 15:04:24: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
    + 15:04:24: ctdb_start:614: status='Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0'
    + 15:04:24: ctdb_start:615: '[' 0 -ne 0 ']'
    + 15:04:24: ctdb_start:621: echo 'Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0'
    + 15:04:24: ctdb_start:621: grep -qs 'UNHEALTHY (THIS'
    + 15:04:24: ctdb_start:609: true
    + 15:04:24: ctdb_start:613: sleep 1
    ++ 15:04:25: ctdb_start:614: invoke_ctdb status
    ++ 15:04:25: invoke_ctdb:371: local timeout
    ++ 15:04:25: invoke_ctdb:372: local timelimit
    ++ 15:04:25: invoke_ctdb:373: timeout=3
    ++ 15:04:25: invoke_ctdb:374: timelimit=120
    ++ 15:04:25: invoke_ctdb:376: '[' -n 90000 ']'
    ++ 15:04:25: invoke_ctdb:377: timeout=90
    ++ 15:04:25: invoke_ctdb:378: timelimit=90
    ++ 15:04:25: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
    + 15:04:25: ctdb_start:614: status='Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0'
    + 15:04:25: ctdb_start:615: '[' 0 -ne 0 ']'
    + 15:04:25: ctdb_start:621: echo 'Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0'
    + 15:04:25: ctdb_start:621: grep -qs 'UNHEALTHY (THIS'
    + 15:04:25: ctdb_start:609: true
    + 15:04:25: ctdb_start:613: sleep 1
    ++ 15:04:26: ctdb_start:614: invoke_ctdb status
    ++ 15:04:26: invoke_ctdb:371: local timeout
    ++ 15:04:26: invoke_ctdb:372: local timelimit
    ++ 15:04:26: invoke_ctdb:373: timeout=3
    ++ 15:04:26: invoke_ctdb:374: timelimit=120
    ++ 15:04:26: invoke_ctdb:376: '[' -n 90000 ']'
    ++ 15:04:26: invoke_ctdb:377: timeout=90
    ++ 15:04:26: invoke_ctdb:378: timelimit=90
    ++ 15:04:26: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
    + 15:04:26: ctdb_start:614: status='Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0'
    + 15:04:26: ctdb_start:615: '[' 0 -ne 0 ']'
    + 15:04:26: ctdb_start:621: echo 'Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0'
    + 15:04:26: ctdb_start:621: grep -qs 'UNHEALTHY (THIS'
    + 15:04:26: ctdb_start:609: true
    + 15:04:26: ctdb_start:613: sleep 1
    ++ 15:04:27: ctdb_start:614: invoke_ctdb status
    ++ 15:04:27: invoke_ctdb:371: local timeout
    ++ 15:04:27: invoke_ctdb:372: local timelimit
    ++ 15:04:27: invoke_ctdb:373: timeout=3
    ++ 15:04:27: invoke_ctdb:374: timelimit=120
    ++ 15:04:27: invoke_ctdb:376: '[' -n 90000 ']'
    ++ 15:04:27: invoke_ctdb:377: timeout=90
    ++ 15:04:27: invoke_ctdb:378: timelimit=90
    ++ 15:04:27: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
    + 15:04:27: ctdb_start:614: status='Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0'
    + 15:04:27: ctdb_start:615: '[' 0 -ne 0 ']'
    + 15:04:27: ctdb_start:621: echo 'Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0'
    + 15:04:27: ctdb_start:621: grep -qs 'UNHEALTHY (THIS'
    + 15:04:27: ctdb_start:609: true
    + 15:04:27: ctdb_start:613: sleep 1
    ++ 15:04:28: ctdb_start:614: invoke_ctdb status
    ++ 15:04:28: invoke_ctdb:371: local timeout
    ++ 15:04:28: invoke_ctdb:372: local timelimit
    ++ 15:04:28: invoke_ctdb:373: timeout=3
    ++ 15:04:28: invoke_ctdb:374: timelimit=120
    ++ 15:04:28: invoke_ctdb:376: '[' -n 90000 ']'
    ++ 15:04:28: invoke_ctdb:377: timeout=90
    ++ 15:04:28: invoke_ctdb:378: timelimit=90
    ++ 15:04:28: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
    + 15:04:28: ctdb_start:614: status='Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0'
    + 15:04:28: ctdb_start:615: '[' 0 -ne 0 ']'
    + 15:04:28: ctdb_start:621: echo 'Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0'
    + 15:04:28: ctdb_start:621: grep -qs 'UNHEALTHY (THIS'
    + 15:04:28: ctdb_start:609: true
    + 15:04:28: ctdb_start:613: sleep 1
    ++ 15:04:29: ctdb_start:614: invoke_ctdb status
    ++ 15:04:29: invoke_ctdb:371: local timeout
    ++ 15:04:29: invoke_ctdb:372: local timelimit
    ++ 15:04:29: invoke_ctdb:373: timeout=3
    ++ 15:04:29: invoke_ctdb:374: timelimit=120
    ++ 15:04:29: invoke_ctdb:376: '[' -n 90000 ']'
    ++ 15:04:29: invoke_ctdb:377: timeout=90
    ++ 15:04:29: invoke_ctdb:378: timelimit=90
    ++ 15:04:29: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
    + 15:04:29: ctdb_start:614: status='Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0'
    + 15:04:29: ctdb_start:615: '[' 0 -ne 0 ']'
    + 15:04:29: ctdb_start:621: echo 'Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0'
    + 15:04:29: ctdb_start:621: grep -qs 'UNHEALTHY (THIS'
    + 15:04:29: ctdb_start:609: true
    + 15:04:29: ctdb_start:613: sleep 1
    ++ 15:04:30: ctdb_start:614: invoke_ctdb status
    ++ 15:04:30: invoke_ctdb:371: local timeout
    ++ 15:04:30: invoke_ctdb:372: local timelimit
    ++ 15:04:30: invoke_ctdb:373: timeout=3
    ++ 15:04:30: invoke_ctdb:374: timelimit=120
    ++ 15:04:30: invoke_ctdb:376: '[' -n 90000 ']'
    ++ 15:04:30: invoke_ctdb:377: timeout=90
    ++ 15:04:30: invoke_ctdb:378: timelimit=90
    ++ 15:04:30: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
    + 15:04:30: ctdb_start:614: status='Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0'
    + 15:04:30: ctdb_start:615: '[' 0 -ne 0 ']'
    + 15:04:30: ctdb_start:621: echo 'Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0'
    + 15:04:30: ctdb_start:621: grep -qs 'UNHEALTHY (THIS'
    + 15:04:30: ctdb_start:609: true
    + 15:04:30: ctdb_start:613: sleep 1
    ++ 15:04:31: ctdb_start:614: invoke_ctdb status
    ++ 15:04:31: invoke_ctdb:371: local timeout
    ++ 15:04:31: invoke_ctdb:372: local timelimit
    ++ 15:04:31: invoke_ctdb:373: timeout=3
    ++ 15:04:31: invoke_ctdb:374: timelimit=120
    ++ 15:04:31: invoke_ctdb:376: '[' -n 90000 ']'
    ++ 15:04:31: invoke_ctdb:377: timeout=90
    ++ 15:04:31: invoke_ctdb:378: timelimit=90
    ++ 15:04:31: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
    + 15:04:31: ctdb_start:614: status='Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0'
    + 15:04:31: ctdb_start:615: '[' 0 -ne 0 ']'
    + 15:04:31: ctdb_start:621: echo 'Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0'
    + 15:04:31: ctdb_start:621: grep -qs 'UNHEALTHY (THIS'
    + 15:04:31: ctdb_start:609: true
    + 15:04:31: ctdb_start:613: sleep 1
    ++ 15:04:32: ctdb_start:614: invoke_ctdb status
    ++ 15:04:32: invoke_ctdb:371: local timeout
    ++ 15:04:32: invoke_ctdb:372: local timelimit
    ++ 15:04:32: invoke_ctdb:373: timeout=3
    ++ 15:04:32: invoke_ctdb:374: timelimit=120
    ++ 15:04:32: invoke_ctdb:376: '[' -n 90000 ']'
    ++ 15:04:32: invoke_ctdb:377: timeout=90
    ++ 15:04:32: invoke_ctdb:378: timelimit=90
    ++ 15:04:32: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
    + 15:04:32: ctdb_start:614: status='Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0'
    + 15:04:32: ctdb_start:615: '[' 0 -ne 0 ']'
    + 15:04:32: ctdb_start:621: echo 'Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0'
    + 15:04:32: ctdb_start:621: grep -qs 'UNHEALTHY (THIS'
    + 15:04:32: ctdb_start:609: true
    + 15:04:32: ctdb_start:613: sleep 1
    ++ 15:04:33: ctdb_start:614: invoke_ctdb status
    ++ 15:04:33: invoke_ctdb:371: local timeout
    ++ 15:04:33: invoke_ctdb:372: local timelimit
    ++ 15:04:33: invoke_ctdb:373: timeout=3
    ++ 15:04:33: invoke_ctdb:374: timelimit=120
    ++ 15:04:33: invoke_ctdb:376: '[' -n 90000 ']'
    ++ 15:04:33: invoke_ctdb:377: timeout=90
    ++ 15:04:33: invoke_ctdb:378: timelimit=90
    ++ 15:04:33: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
    + 15:04:33: ctdb_start:614: status='Number of nodes:2
    pnn:0 172.30.199.31 OK (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0'
    + 15:04:33: ctdb_start:615: '[' 0 -ne 0 ']'
    + 15:04:33: ctdb_start:621: echo 'Number of nodes:2
    pnn:0 172.30.199.31 OK (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0'
    + 15:04:33: ctdb_start:621: grep -qs 'UNHEALTHY (THIS'
    + 15:04:33: ctdb_start:625: set_ctdb_variables
    + 15:04:33: set_ctdb_variables:416: rv=0
    + 15:04:33: set_ctdb_variables:417: set
    + 15:04:33: set_ctdb_variables:417: grep '^CTDB_SET_'
    + 15:04:33: set_ctdb_variables:417: cut -d_ -f3-
    + 15:04:33: set_ctdb_variables:418: read v
    + 15:04:33: set_ctdb_variables:423: return 0
    + 15:04:33: ctdb_start:626: return 0
    + 15:04:33: main:772: rc=0
    + 15:04:33: main:773: ocf_log debug 'res_CTDB start : 0'
    + 15:04:33: ocf_log:322: '[' 2 -lt 2 ']'
    + 15:04:33: ocf_log:326: __OCF_PRIO=debug
    + 15:04:33: ocf_log:327: shift
    + 15:04:33: ocf_log:328: __OCF_MSG='res_CTDB start : 0'
    + 15:04:33: ocf_log:330: case "${__OCF_PRIO}" in
    + 15:04:33: ocf_log:335: __OCF_PRIO=DEBUG
    + 15:04:33: ocf_log:339: '[' DEBUG = DEBUG ']'
    + 15:04:33: ocf_log:340: ha_debug 'DEBUG: res_CTDB start : 0'
    + 15:04:33: ha_debug:259: '[' x0 = x0 ']'
    + 15:04:33: ha_debug:260: return 0
    + 15:04:33: main:774: exit 0

  2. #2

    Re: HA Resource for CTDB fails after Upgrade to SP4

    geidelt,

    It appears that in the past few days you have not received a response to your
    posting. That concerns us, and has triggered this automated reply.

    These forums are peer-to-peer, best effort, volunteer run and that if your issue
    is urgent or not getting a response, you might try one of the following options:

    - Visit http://www.suse.com/support and search the knowledgebase and/or check all
    the other support options available.
    - Open a service request: https://www.suse.com/support
    - You could also try posting your message again. Make sure it is posted in the
    correct newsgroup. (http://forums.suse.com)

    Be sure to read the forum FAQ about what to expect in the way of responses:
    http://forums.suse.com/faq.php

    If this is a reply to a duplicate posting or otherwise posted in error, please
    ignore and accept our apologies and rest assured we will issue a stern reprimand
    to our posting bot..

    Good luck!

    Your SUSE Forums Team
    http://forums.suse.com



Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •