# Rancher HA Management Stack not coming up **URL:** \ **Category:** Rancher 1.x\ **Created:** [April 15, 2016, 10:03am UTC](https://forums.suse.com/t/rancher-ha-management-stack-not-coming-up/2466 "2016-04-15T10:03:14Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![shethchintan7](https://avatars.discourse-cdn.com/v4/letter/s/848f3c/32.png) [@shethchintan7](https://forums.suse.com/u/shethchintan7)\ **Post date:** [April 15, 2016, 10:03am UTC](https://forums.suse.com/t/rancher-ha-management-stack-not-coming-up/2466/1 "2016-04-15T10:03:14Z")
I am trying out Rancher v1.0.1 HA Setup on AWS OS : RancherOS Using RDS as the external Mysql instance and I’m running the script on 3 nodes One instance of both go-machine-service & rancher-compose-executor are not starting and thus my management stack. Any help would be much appreciated ---
**Author:** ![Rucknar](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/rucknar/32/107_2.png) [@Rucknar](https://forums.suse.com/u/Rucknar)\ **Post date:** [April 15, 2016, 12:58pm UTC](https://forums.suse.com/t/rancher-ha-management-stack-not-coming-up/2466/2 "2016-04-15T12:58:02Z")
Could you get the logs for those containers that won’t start and post them here. ---
**Author:** ![shethchintan7](https://avatars.discourse-cdn.com/v4/letter/s/848f3c/32.png) [@shethchintan7](https://forums.suse.com/u/shethchintan7)\ **Post date:** [April 15, 2016, 1:55pm UTC](https://forums.suse.com/t/rancher-ha-management-stack-not-coming-up/2466/3 "2016-04-15T13:55:56Z")
They never reach running status. Initializing \> Reconciling \> Stopping so not sure how do I find the logs for failing instance Last log in docker logs -f rancher-ha is the following on all three nodes time=“2016-04-15T09:50:02Z” level=info msg=“Container agent is not running in state &types.ContainerState{Status:“exited”, Running:false, Paused:false, Restarting:false, OOMKilled:false, Dead:false, Pid:0, ExitCode:0, Error:”", StartedAt:“2016-04-15T09:46:52.499514335Z”, FinishedAt:“2016-04-15T09:47:02.105292501Z”}" component=docker time=“2016-04-15T09:50:02Z” level=info msg=“Deleting container 89f5a53bd7ce3b263891f9d303fadd052b29edd048c9fe6af269151035893b06” component=docker /var/log/docker.log is spamming time=“2016-04-15T13:53:46.124242130Z” level=error msg=“Handler for GET /v1.22/containers/rancher-ha-agent/json returned error: No such container: rancher-ha-agent” ---
**Author:** ![Rucknar](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/rucknar/32/107_2.png) [@Rucknar](https://forums.suse.com/u/Rucknar)\ **Post date:** [April 15, 2016, 2:14pm UTC](https://forums.suse.com/t/rancher-ha-management-stack-not-coming-up/2466/4 "2016-04-15T14:14:35Z")
I’m guessing here… Have a look at the logs coming out of the cattle container, it might not log to STDOUT though. Think it’s logs are in `/var/lib/cattle/logs` ---
**Author:** ![shethchintan7](https://avatars.discourse-cdn.com/v4/letter/s/848f3c/32.png) [@shethchintan7](https://forums.suse.com/u/shethchintan7)\ **Post date:** [April 20, 2016, 7:20am UTC](https://forums.suse.com/t/rancher-ha-management-stack-not-coming-up/2466/5 "2016-04-20T07:20:29Z")
So that issue is fixed now, there were some loadbalancer issues. But I’ve hit another snag So management stack is working great now but there’s a networking problem in the hosts that I’m adding. Containers are not able to communicate among each other, basically I cannot ping a container from any other worker node. Is there a different method for adding worker hosts to the HA stack? Network agent has following in STDERR RTNETLINK answers: No such file or directory SIOCSARP: Invalid argument arp: cannot set entry on line 2 of etherfile content-home/etc/cattle/ethers ! ---
**Author:** ![Rucknar](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/rucknar/32/107_2.png) [@Rucknar](https://forums.suse.com/u/Rucknar)\ **Post date:** [April 20, 2016, 7:30am UTC](https://forums.suse.com/t/rancher-ha-management-stack-not-coming-up/2466/6 "2016-04-20T07:30:22Z")
Seeing similar actually: ```auto 4/19/2016 10:29:46 PMSIOCSARP: Invalid argument 4/19/2016 10:29:46 PMarp: cannot set entry on line 2 of etherfile content-home/etc/cattle/ethers ! ``` Should probably raise as a bug on github. ---
**Author:** ![shethchintan7](https://avatars.discourse-cdn.com/v4/letter/s/848f3c/32.png) [@shethchintan7](https://forums.suse.com/u/shethchintan7)\ **Post date:** [April 20, 2016, 7:32am UTC](https://forums.suse.com/t/rancher-ha-management-stack-not-coming-up/2466/7 "2016-04-20T07:32:47Z")
Also one more weird thing, Host ips on the UI are not correct. Nodes on which HA is running they have correct IPs but any host that I add get an incorrect one. ---
**Author:** ![Rucknar](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/rucknar/32/107_2.png) [@Rucknar](https://forums.suse.com/u/Rucknar)\ **Post date:** [April 20, 2016, 8:15am UTC](https://forums.suse.com/t/rancher-ha-management-stack-not-coming-up/2466/8 "2016-04-20T08:15:46Z")
Are they all the same IP? just noticed that myself! I think it’s breaking the host\>host communication as it’s establishing the tunnel to the wrong IP ---
**Author:** ![shethchintan7](https://avatars.discourse-cdn.com/v4/letter/s/848f3c/32.png) [@shethchintan7](https://forums.suse.com/u/shethchintan7)\ **Post date:** [April 20, 2016, 8:52am UTC](https://forums.suse.com/t/rancher-ha-management-stack-not-coming-up/2466/9 "2016-04-20T08:52:54Z")
They are not the same in my case ---
**Author:** ![Rucknar](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/rucknar/32/107_2.png) [@Rucknar](https://forums.suse.com/u/Rucknar)\ **Post date:** [April 20, 2016, 9:28am UTC](https://forums.suse.com/t/rancher-ha-management-stack-not-coming-up/2466/10 "2016-04-20T09:28:08Z")
Just validated, you don’t get this issue when not using HA. ---
**Author:** ![shethchintan7](https://avatars.discourse-cdn.com/v4/letter/s/848f3c/32.png) [@shethchintan7](https://forums.suse.com/u/shethchintan7)\ **Post date:** [April 20, 2016, 9:42am UTC](https://forums.suse.com/t/rancher-ha-management-stack-not-coming-up/2466/11 "2016-04-20T09:42:11Z")
Yeap, same here. HA issue it seems ---
**Author:** ![shethchintan7](https://avatars.discourse-cdn.com/v4/letter/s/848f3c/32.png) [@shethchintan7](https://forums.suse.com/u/shethchintan7)\ **Post date:** [April 20, 2016, 10:57am UTC](https://forums.suse.com/t/rancher-ha-management-stack-not-coming-up/2466/12 "2016-04-20T10:57:03Z")
So it seems it is taking ELB IPs ( private ip ) ELB Private ips can be found by > aws ec2 describe-network-interfaces --filters “Name=description,Values=ELB rancher” |grep -wE ‘Description|PrivateIpAddress’ I used -e CATTLE\_AGENT\_IP=‘’ to explicitly state IP and it seems to work. ---
**Author:** ![Rucknar](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/rucknar/32/107_2.png) [@Rucknar](https://forums.suse.com/u/Rucknar)\ **Post date:** [April 20, 2016, 4:17pm UTC](https://forums.suse.com/t/rancher-ha-management-stack-not-coming-up/2466/13 "2016-04-20T16:17:44Z")
Good find. So Rancher is possibly doing a reverse DNS lookup on the Callback URL and that value is somehow being translated into the host IP address. Seems strange as it works on a single server without ELB. ---
**Author:** ![Rucknar](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/rucknar/32/107_2.png) [@Rucknar](https://forums.suse.com/u/Rucknar)\ **Post date:** [April 21, 2016, 5:06pm UTC](https://forums.suse.com/t/rancher-ha-management-stack-not-coming-up/2466/14 "2016-04-21T17:06:20Z")
So, enabling the Proxy protocol on the ELB i’m led to believe might fix this issue. I’ve not had success personally but it’s worth a shot if you get chance. ---
**Author:** ![Fernando\_Felicissimo](https://avatars.discourse-cdn.com/v4/letter/f/76d3ee/32.png) [@Fernando\_Felicissimo](https://forums.suse.com/u/Fernando_Felicissimo)\ **Post date:** [April 21, 2016, 5:37pm UTC](https://forums.suse.com/t/rancher-ha-management-stack-not-coming-up/2466/15 "2016-04-21T17:37:13Z")
I’m having the same problem … and some container just does not start up and can not get “network agent” via shell to make a ping between them … and a bug you? Do you have a fix? regards, ---
**Author:** ![shethchintan7](https://avatars.discourse-cdn.com/v4/letter/s/848f3c/32.png) [@shethchintan7](https://forums.suse.com/u/shethchintan7)\ **Post date:** [April 22, 2016, 1:17am UTC](https://forums.suse.com/t/rancher-ha-management-stack-not-coming-up/2466/16 "2016-04-22T01:17:07Z")
Check the logs for exited containers. I had two main issues, https certificates not matching URL and elb listerners being http https, it should be tcp and ssl. ---
**Author:** ![denise](https://avatars.discourse-cdn.com/v4/letter/d/82dd89/32.png) [@denise](https://forums.suse.com/u/denise)\ **Post date:** [April 29, 2016, 8:36pm UTC](https://forums.suse.com/t/rancher-ha-management-stack-not-coming-up/2466/17 "2016-04-29T20:36:18Z")
@shethchintan7 Are you still having issues? We have started to recommend specifying the `-e CATTLE_AGENT_IP` when using an AWS setup with ELB. ---
**Author:** ![ThatsNinja](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/thatsninja/32/1085_2.png) [@ThatsNinja](https://forums.suse.com/u/ThatsNinja)\ **Post date:** [April 29, 2016, 8:38pm UTC](https://forums.suse.com/t/rancher-ha-management-stack-not-coming-up/2466/18 "2016-04-29T20:38:44Z")
@denise What are you recommending setting the `CATTLE_AGENT_IP` to? ---
**Author:** ![shethchintan7](https://avatars.discourse-cdn.com/v4/letter/s/848f3c/32.png) [@shethchintan7](https://forums.suse.com/u/shethchintan7)\ **Post date:** [April 30, 2016, 1:49am UTC](https://forums.suse.com/t/rancher-ha-management-stack-not-coming-up/2466/19 "2016-04-30T01:49:16Z")
@denise No, it’s working great. Thanks ---
**Author:** ![denise](https://avatars.discourse-cdn.com/v4/letter/d/82dd89/32.png) [@denise](https://forums.suse.com/u/denise)\ **Post date:** [April 30, 2016, 2:10am UTC](https://forums.suse.com/t/rancher-ha-management-stack-not-coming-up/2466/20 "2016-04-30T02:10:04Z")
@ThatsNinja when adding the host, you set it it to the IP of the host. These docs explain what you are doing when setting the `CATTLE_AGENT_IP` but just a different known use case of when it needs to be set. [http://docs.rancher.com/rancher/rancher-ui/infrastructure/hosts/custom/#adding-hosts-to-the-same-machine-as-rancher-server](http://docs.rancher.com/rancher/rancher-ui/infrastructure/hosts/custom/#adding-hosts-to-the-same-machine-as-rancher-server) [Next page](https://forums.suse.com/t/rancher-ha-management-stack-not-coming-up/2466.md?page=2)