# Automatically remove disconnected hosts **URL:** \ **Category:** Rancher 1.x\ **Created:** [November 14, 2015, 9:33pm UTC](https://forums.suse.com/t/automatically-remove-disconnected-hosts/956 "2015-11-14T21:33:41Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![tbossert](https://avatars.discourse-cdn.com/v4/letter/t/ac91a4/32.png) [@tbossert](https://forums.suse.com/u/tbossert)\ **Post date:** [November 14, 2015, 9:33pm UTC](https://forums.suse.com/t/automatically-remove-disconnected-hosts/956/1 "2015-11-14T21:33:41Z")
I am using AWS autoscaling to bring up hosts that register with rancher on boot, however when the machines are shutdown after load decreases they end up in a reconnecting state and traffic from rancher LB’s seem to continue to flow to the containers on these instances until I manually remove them in the UI. This caused a period of outage as some of the requests coming in would time out. The first thing that needs to change is that as soon as a host goes into reconnecting all traffic should cease to those containers, all LB’s,service aliases should be aware of this. The second thing which would be a nice to have is the option in rancher to say, any host that is disconnected for more than a specified configureable timeout should be deactivated and purged. Please let me know if there are any workarounds or things I could try to mitigate the outages when spinning down instances. Thanks - Trevor ---
**Author:** ![ColtonProvias](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/coltonprovias/32/504_2.png) [@ColtonProvias](https://forums.suse.com/u/ColtonProvias)\ **Post date:** [November 14, 2015, 10:59pm UTC](https://forums.suse.com/t/automatically-remove-disconnected-hosts/956/2 "2015-11-14T22:59:19Z")
One workaround may be to write a script that periodically hits the API to delete inactive hosts from Rancher. ---
**Author:** ![vincent](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/vincent/32/7156_2.png) [@vincent](https://forums.suse.com/u/vincent)\ **Post date:** [November 14, 2015, 11:01pm UTC](https://forums.suse.com/t/automatically-remove-disconnected-hosts/956/3 "2015-11-14T23:01:47Z")
Integration to be able to have the host automatically unregister itself in response to a spot instance shutdown event is coming soon… ---
**Author:** ![tbossert](https://avatars.discourse-cdn.com/v4/letter/t/ac91a4/32.png) [@tbossert](https://forums.suse.com/u/tbossert)\ **Post date:** [November 14, 2015, 11:04pm UTC](https://forums.suse.com/t/automatically-remove-disconnected-hosts/956/4 "2015-11-14T23:04:02Z")
Yea for deleting hosts, I was going to write a script until the functionality is added into Rancher. In all reality the issue with traffic seeming to still go to disconnected hosts is my most immediate problem. ---
**Author:** ![tbossert](https://avatars.discourse-cdn.com/v4/letter/t/ac91a4/32.png) [@tbossert](https://forums.suse.com/u/tbossert)\ **Post date:** [November 14, 2015, 11:04pm UTC](https://forums.suse.com/t/automatically-remove-disconnected-hosts/956/5 "2015-11-14T23:04:19Z")
Awesome, Thanks for letting me know ---
**Author:** ![vincent](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/vincent/32/7156_2.png) [@vincent](https://forums.suse.com/u/vincent)\ **Post date:** [November 15, 2015, 12:06am UTC](https://forums.suse.com/t/automatically-remove-disconnected-hosts/956/6 "2015-11-15T00:06:58Z")
Is there a health check in the service(s) being balanced to? If there is the containers on failed hosts should stop getting traffic from haproxy quickly (unhealthy threshold \* timeout). ---
**Author:** ![tbossert](https://avatars.discourse-cdn.com/v4/letter/t/ac91a4/32.png) [@tbossert](https://forums.suse.com/u/tbossert)\ **Post date:** [November 15, 2015, 12:14am UTC](https://forums.suse.com/t/automatically-remove-disconnected-hosts/956/7 "2015-11-15T00:14:52Z")
Yes I do have health checks on the services http check with settings of 3 failed checks at interval of 2 seconds The behavior I saw earlier today seemed to only be resolved after I manually removed the disconnected hosts through the UI, before that for several minutes I had intermittent failures in reaching my app. The app and load balancer are set to be global in case that makes and difference. I will be doing some testing in our staging env next week and will hopefully have more info at that time. Thanks for the quick responses. ---
**Author:** ![tbossert](https://avatars.discourse-cdn.com/v4/letter/t/ac91a4/32.png) [@tbossert](https://forums.suse.com/u/tbossert)\ **Post date:** [November 16, 2015, 9:16pm UTC](https://forums.suse.com/t/automatically-remove-disconnected-hosts/956/8 "2015-11-16T21:16:08Z")
Part of what I am seeing might be not so much the load balancers not updating, but when I lose a host with a load balancer container on it, the list of load balancer ips is not being updated. This results in traffic flowing to load balancer containers that no longer exist. ---
**Author:** ![vincent](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/vincent/32/7156_2.png) [@vincent](https://forums.suse.com/u/vincent)\ **Post date:** [November 17, 2015, 12:28am UTC](https://forums.suse.com/t/automatically-remove-disconnected-hosts/956/9 "2015-11-17T00:28:01Z")
Ah yes… If you’ve got your own DNS entries listing the host IPs then there’s no way for us to remove them from there. What some people have done before is put something like Amazon ELB as a ‘dumb’ TCP balancer in front of the Rancher one, so it can automatically remove failing IPs. In the last release we added the Route53 DNS catalog item which can automatically keep a Route53 zone up to date with the active host IPs for each service (i.e. `serviceName.stackName.environmentName.yourzone.com`), and then you can CNAME that to the actual names you want to balance. There are some bugs in that release with handling disconnected hosts though which you might run in to, those will be fixed in the next release (0.46). ---
**Author:** ![tbossert](https://avatars.discourse-cdn.com/v4/letter/t/ac91a4/32.png) [@tbossert](https://forums.suse.com/u/tbossert)\ **Post date:** [November 18, 2015, 4:33am UTC](https://forums.suse.com/t/automatically-remove-disconnected-hosts/956/10 "2015-11-18T04:33:02Z")
Could you point me to the github issues for the those bugs you mentioned. It would be helpful to know what to expect. Thanks - Trevor ---
**Author:** ![vincent](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/vincent/32/7156_2.png) [@vincent](https://forums.suse.com/u/vincent)\ **Post date:** [November 18, 2015, 4:57pm UTC](https://forums.suse.com/t/automatically-remove-disconnected-hosts/956/11 "2015-11-18T16:57:52Z")
Something like this… [https://github.com/rancher/rancher/issues?q=milestone%3A"Milestone+11%2F11%2F2015"+is%3Aclosed+health](https://github.com/rancher/rancher/issues?q=milestone%3A%22Milestone+11%2F11%2F2015%22+is%3Aclosed+health) (and 0.46 is out now) ---
**Author:** ![tbossert](https://avatars.discourse-cdn.com/v4/letter/t/ac91a4/32.png) [@tbossert](https://forums.suse.com/u/tbossert)\ **Post date:** [November 18, 2015, 5:03pm UTC](https://forums.suse.com/t/automatically-remove-disconnected-hosts/956/12 "2015-11-18T17:03:56Z")
Awesome I am upgrading as I type this. Thanks - Trevor ---
**Author:** ![tbossert](https://avatars.discourse-cdn.com/v4/letter/t/ac91a4/32.png) [@tbossert](https://forums.suse.com/u/tbossert)\ **Post date:** [April 12, 2016, 5:12pm UTC](https://forums.suse.com/t/automatically-remove-disconnected-hosts/956/13 "2016-04-12T17:12:24Z")
Was curious if there was an updated timeline for host removal on spot instance shutdown? ---
**Author:** ![denise](https://avatars.discourse-cdn.com/v4/letter/d/82dd89/32.png) [@denise](https://forums.suse.com/u/denise)\ **Post date:** [April 28, 2016, 5:03pm UTC](https://forums.suse.com/t/automatically-remove-disconnected-hosts/956/14 "2016-04-28T17:03:44Z")
It exists, but we haven’t actually declared support or documentation for it. Could you add an issue in Github so that we could track it to make it more official? ---
**Author:** ![jmreicha](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/jmreicha/32/444_2.png) [@jmreicha](https://forums.suse.com/u/jmreicha)\ **Post date:** [May 17, 2016, 6:15am UTC](https://forums.suse.com/t/automatically-remove-disconnected-hosts/956/15 "2016-05-17T06:15:46Z")
Interested in this functionality as well. Is there an issue to follow for this or somewhere to go look at examples? ---
**Author:** ![ryanwalls](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/ryanwalls/32/2762_2.png) [@ryanwalls](https://forums.suse.com/u/ryanwalls)\ **Post date:** [March 7, 2017, 9:01pm UTC](https://forums.suse.com/t/automatically-remove-disconnected-hosts/956/16 "2017-03-07T21:01:52Z")
@denise @vincent Any updates on unregistering a host? I’m experiencing lots of networking issues when we scale in hosts in my k8s cluster. [https://github.com/rancher/rancher/issues/8086](https://github.com/rancher/rancher/issues/8086) ---
**Author:** ![ampedandwired](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/ampedandwired/32/1721_2.png) [@ampedandwired](https://forums.suse.com/u/ampedandwired)\ **Post date:** [March 8, 2017, 1:52am UTC](https://forums.suse.com/t/automatically-remove-disconnected-hosts/956/17 "2017-03-08T01:52:05Z")
I wrote a [Rancher reaper](https://hub.docker.com/r/ampedandwired/rancher-reaper/) service that periodically purges Rancher disconnected hosts that no longer exist in AWS. You could give that a try while waiting for an officially endorsed solution. ---
**Author:** ![wmbutler](https://avatars.discourse-cdn.com/v4/letter/w/aca169/32.png) [@wmbutler](https://forums.suse.com/u/wmbutler)\ **Post date:** [April 30, 2017, 12:20am UTC](https://forums.suse.com/t/automatically-remove-disconnected-hosts/956/18 "2017-04-30T00:20:22Z")
I wrote [wmbutler/rancher-purge](https://hub.docker.com/r/wmbutler/rancher-purge/). It’s a container with configurable Environment variables. Should take no longer than a few minutes to configure and deploy.