# Upgrading Rancher HA

**URL:** <https://forums.suse.com/t/upgrading-rancher-ha/2899>\
**Category:** Rancher 1.x\
**Created:** [May 24, 2016, 8:53pm UTC](https://forums.suse.com/t/upgrading-rancher-ha/2899 "2016-05-24T20:53:10Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![whiteadam](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/whiteadam/32/493_2.png) [@whiteadam](https://forums.suse.com/u/whiteadam)\
**Post date:** [May 24, 2016, 8:53pm UTC](https://forums.suse.com/t/upgrading-rancher-ha/2899/1 "2016-05-24T20:53:10Z")

</div>

I’m trying to upgrade Rancher Server HA to the latest version, but it appears that it doesn’t work like the docs for a single server (or like I assumed, which was just re-launching the service with the latest server image version).

Does anyone know the proper upgrade path for HA? Maybe @denise since she pined the docs for the single server version 🙂

Thanks in advance,  
-Adam

---

<div class="post-metadata">

**Author:** ![vincent](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/vincent/32/7156_2.png) [@vincent](https://forums.suse.com/u/vincent)\
**Post date:** [May 25, 2016, 5:55am UTC](https://forums.suse.com/t/upgrading-rancher-ha/2899/2 "2016-05-25T05:55:43Z")

</div>

Re-run the HA setup script on each host with a newer image tag; The script is saved on each host, so

`/var/lib/rancher/bin/rancher-ha-start.sh rancher/server:v1.1.0-dev2`

---

<div class="post-metadata">

**Author:** ![whiteadam](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/whiteadam/32/493_2.png) [@whiteadam](https://forums.suse.com/u/whiteadam)\
**Post date:** [May 25, 2016, 6:17pm UTC](https://forums.suse.com/t/upgrading-rancher-ha/2899/3 "2016-05-25T18:17:31Z")

</div>

Thanks Vincent,

I tried that, but now it’s spewing out things like:

```auto
time="2016-05-25T18:16:21Z" level=info msg="[0/10] [tunnel]: Starting "
time="2016-05-25T18:16:21Z" level=info msg="Upgrading tunnel"
time="2016-05-25T18:16:21Z" level=error msg="Failed Starting tunnel : Bad response statusCode [422]. Status [422 Unprocessable Entity]. Body: [code=InvalidAction, fieldName=Can't upgrade selector only service] from [http://172.17.0.1:18080/v1/projects/1a8/services/1s2/?action=upgrade]"

```

Any ideas?

---

<div class="post-metadata">

**Author:** ![whiteadam](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/whiteadam/32/493_2.png) [@whiteadam](https://forums.suse.com/u/whiteadam)\
**Post date:** [May 25, 2016, 11:14pm UTC](https://forums.suse.com/t/upgrading-rancher-ha/2899/4 "2016-05-25T23:14:54Z")

</div>

Don’t hurt yourself on this one, I ended up wiping out the DB since I had nothing on it and starting from scratch. The good news is, HA ELB with SSL Offloading works, weee.

---

<div class="post-metadata">

**Author:** ![Matt\_Welch](https://avatars.discourse-cdn.com/v4/letter/m/a587f6/32.png) [@Matt\_Welch](https://forums.suse.com/u/Matt_Welch)\
**Post date:** [May 26, 2016, 11:51am UTC](https://forums.suse.com/t/upgrading-rancher-ha/2899/5 "2016-05-26T11:51:20Z")

</div>

Do you think you can share a little more information about your setup? I’ve been trying to stand up a very basic 3 node Rancher HA installation in AWS, but have run into problem after problem. A few of the issues I’ve run into are here: [Production Rancher HA on AWS](http://forums.suse.com/t/production-rancher-ha-on-aws/2863)

You appear to have been successful and I don’t mind starting over from scratch as I’ve already done that a half dozen times anyway trying to get this working.

For instance, what AMI are you starting with? Docker version? Anything else that might get us off the starting block…

---

<div class="post-metadata">

**Author:** ![aemneina](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/aemneina/32/1323_2.png) [@aemneina](https://forums.suse.com/u/aemneina)\
**Post date:** [May 27, 2016, 9:34pm UTC](https://forums.suse.com/t/upgrading-rancher-ha/2899/6 "2016-05-27T21:34:32Z")

</div>

I’ve got my AWS + ELB + SSL working. I’ve found this post accurate in the steps to get this going:

[https://github.com/rancher/rancher/issues/4634#issuecomment-216246874](https://github.com/rancher/rancher/issues/4634#issuecomment-216246874)

---

<div class="post-metadata">

**Author:** ![alexR](https://avatars.discourse-cdn.com/v4/letter/a/8dc957/32.png) [@alexR](https://forums.suse.com/u/alexR)\
**Post date:** [June 27, 2016, 11:46am UTC](https://forums.suse.com/t/upgrading-rancher-ha/2899/7 "2016-06-27T11:46:36Z")

</div>

@vincent: you mention that a version upgrade for rancher-server in HA just neets to run:

`/var/lib/rancher/bin/rancher-ha-start.sh rancher/server:v1.1.0-devX`

Do we need to stop the running containers running with the old version? At what moment should I run the script in the _next_ rancher-server in the cluster? or should they be done all at once?

I’ve tried going from dev3 -\> dev4 and dev4 -\> dev5 and I’ve always ran into problems, having to resort to delete all of the docker images and the `/var/lib/rancher/state` directory and running the script above _several_ times before reaching a green state. That is clearly not how the procedure should work, I would really appreciate more details of how you guys at Rancher Labs do it.

Best regards,

Alejandro

---

<div class="post-metadata">

**Author:** ![whiteadam](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/whiteadam/32/493_2.png) [@whiteadam](https://forums.suse.com/u/whiteadam)\
**Post date:** [June 27, 2016, 12:14pm UTC](https://forums.suse.com/t/upgrading-rancher-ha/2899/8 "2016-06-27T12:14:15Z")

</div>

I’ve had the same experience, it’s not as simple as just running a script.

I was doing the same as you with all of the manual stuff, then I got tired of it and put my 3 masters in an ASG and am just terminating them when I need an upgrade (one by one). I can give you my cloud-init script if you’re interested. It may only work on ROS though.

-Adam

---

<div class="post-metadata">

**Author:** ![alexR](https://avatars.discourse-cdn.com/v4/letter/a/8dc957/32.png) [@alexR](https://forums.suse.com/u/alexR)\
**Post date:** [June 27, 2016, 12:29pm UTC](https://forums.suse.com/t/upgrading-rancher-ha/2899/9 "2016-06-27T12:29:43Z")

</div>

Thanks for the fast reply @whiteadam , it sure would be a great help if you could share your cloud-init scripts, if so just to see other approaches, as I am in Debian + Bare Metal.

Cheers,

---

<div class="post-metadata">

**Author:** ![whiteadam](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/whiteadam/32/493_2.png) [@whiteadam](https://forums.suse.com/u/whiteadam)\
**Post date:** [June 27, 2016, 12:56pm UTC](https://forums.suse.com/t/upgrading-rancher-ha/2899/10 "2016-06-27T12:56:10Z")

</div>

Here you go, it’s a bit hacky, but I’m sure you can figure it out 🙂

This may be MUCH easier on Debian, RancherOS has some finicky little things you have to work around, which is why there is an init script to wait 300 secs before executing the HA script.

The formatting here might be a little strange, I tried.

```
#cloud-config
write_files:
  - path: /etc/rc.local
    permissions: "0755"
    owner: root
    content: |
      #!/bin/bash
      sleep 300
      sudo bash /etc/rancher-ha.sh rancher/server:v1.1.0-dev5
  - path: /etc/rancher-ha.sh
    permissions: "0755"
    owner: root
    content: |
      #!/bin/sh
      set -e
      umask 077

      IMAGE=$1
      if ["$IMAGE" = ""]; then
        IMAGE=rancher/server
      fi

      mkdir -p /var/lib/rancher/etc/server
      mkdir -p /var/lib/rancher/etc/ssl
      mkdir -p /var/lib/rancher/bin

      echo Creating /var/lib/rancher/etc/server.conf
      cat > /var/lib/rancher/etc/server.conf << EOF
      export CATTLE_HA_CLUSTER_SIZE=3
      export CATTLE_HA_HOST_REGISTRATION_URL=https://rancher.example.com
      export CATTLE_HA_CONTAINER_PREFIX=rancher-ha-

      export CATTLE_DB_CATTLE_MYSQL_HOST=whatever.us-east-1.rds.amazonaws.com
      export CATTLE_DB_CATTLE_MYSQL_PORT=3306
      export CATTLE_DB_CATTLE_MYSQL_NAME=rancher
      export CATTLE_DB_CATTLE_USERNAME=rancher
      export CATTLE_DB_CATTLE_PASSWORD=<password in base64>

      export CATTLE_HA_PORT_REDIS=6379
      export CATTLE_HA_PORT_SWARM=2376
      export CATTLE_HA_PORT_HTTP=80
      export CATTLE_HA_PORT_HTTPS=443
      export CATTLE_HA_PORT_PP_HTTP=81
      export CATTLE_HA_PORT_PP_HTTPS=444
      export CATTLE_HA_PORT_ZK_CLIENT=2181
      export CATTLE_HA_PORT_ZK_QUORUM=2888
      export CATTLE_HA_PORT_ZK_LEADER=3888

      # Uncomment below to force HA enabled and not require one to set it in the UI
      export CATTLE_HA_ENABLED=true
      EOF

      echo Creating /var/lib/rancher/etc/server/encryption.key
      if [-e /var/lib/rancher/etc/server/encryption.key]; then
        mv /var/lib/rancher/etc/server/encryption.key /var/lib/rancher/etc/server/encryption.key.`date '+%s'`
      fi
      cat > /var/lib/rancher/etc/server/encryption.key << EOF
      eR9azJrestofthekeyhere
      EOF

      echo Creating /var/lib/rancher/bin/rancher-ha-start.sh
      cat > /var/lib/rancher/bin/rancher-ha-start.sh << "EOF"
      #!/bin/sh
      set -e

      IMAGE=$1
      if ["$IMAGE" = ""]; then
        echo Usage: $0 DOCKER_IMAGE
        exit 1
      fi

      docker rm -fv rancher-ha >/dev/null 2>&1 || true
      ID=`docker run --restart=always -d -v /var/run/docker.sock:/var/run/docker.sock --name rancher-ha --net host --privileged -v /var/lib/rancher/etc:/var/lib/rancher/etc $IMAGE ha`

      echo Started container rancher-ha $ID
      echo Run the below to see the logs
      echo
      echo docker logs -f rancher-ha
      EOF

      chmod +x /var/lib/rancher/bin/rancher-ha-start.sh

      echo Running: /var/lib/rancher/bin/rancher-ha-start.sh $IMAGE
      echo To re-run please execute: /var/lib/rancher/bin/rancher-ha-start.sh $IMAGE
      exec /var/lib/rancher/bin/rancher-ha-start.sh $IMAGE
```