# Host-required labels are not applied properly?

**URL:** <https://forums.suse.com/t/host-required-labels-are-not-applied-properly/8182>\
**Category:** General\
**Created:** [January 9, 2018, 9:32am UTC](https://forums.suse.com/t/host-required-labels-are-not-applied-properly/8182 "2018-01-09T09:32:43Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![EugenMayer](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/eugenmayer/32/1415_2.png) [@EugenMayer](https://forums.suse.com/u/EugenMayer)\
**Post date:** [January 9, 2018, 9:32am UTC](https://forums.suse.com/t/host-required-labels-are-not-applied-properly/8182/1 "2018-01-09T09:32:43Z")

</div>

I have an environment with 2 hosts:

a) host-rancher\_internal  
b) host-ci-workers

host-ci-workers has a host-required label: `io.rancher.scheduler.affinity=ci_worker`  
 ![image](https://us1.discourse-cdn.com/flex022/uploads/suse/original/2X/2/28eede6d7a62796d71b4c3c6123282045b9708fe.png)

 ![image](https://us1.discourse-cdn.com/flex022/uploads/suse/original/2X/a/ae8423c45d072f64efc66c32101721af41bc0a5b.png)

while the host-rancher\_internal does not have any required label.  
 ![image](https://us1.discourse-cdn.com/flex022/uploads/suse/original/2X/9/9cf04d2c05891c88ca71612937fcd681ac550c4a.png)

Today, due to the ros 1.1.3 upgrade, i had top restart rancher. What happened is, that also stacks without the label `io.rancher.scheduler.affinity=ci_worker` have been scheduled to host-ci-workers.

Why is that - having no label should not meat the requirement having the label `io.rancher.scheduler.affinity=ci_worker`?

---

<div class="post-metadata">

**Author:** ![vincent](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/vincent/32/7156_2.png) [@vincent](https://forums.suse.com/u/vincent)\
**Post date:** [January 10, 2018, 10:36pm UTC](https://forums.suse.com/t/host-required-labels-are-not-applied-properly/8182/2 "2018-01-10T22:36:13Z")

</div>

The `io.rancher.scheduler.require_any` label (which is what the UI is setting there) is only looked at by the external resource-based scheduler (the catalog item), do you have it deployed? And is it actually running anywhere?

You don’t have `io.rancher.container.system` in the list so if it _was_ working you wouldn’t get any of system services on the host(s), which is unlikely to be desirable.

You should not call your labels `io.rancher.*`. Make up your own prefix or don’t use one at all… But they are namespaced specifically to avoid conflict with arbitrary user entries and by using our prefix you’re asking for trouble. Any future release may decide to take arbitrary actions based on labels in our namespace. And `io.rancher.scheduler.affinity` specifically is as you know already part of the container scheduling labels for services.

---

<div class="post-metadata">

**Author:** ![EugenMayer](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/eugenmayer/32/1415_2.png) [@EugenMayer](https://forums.suse.com/u/EugenMayer)\
**Post date:** [January 11, 2018, 10:06am UTC](https://forums.suse.com/t/host-required-labels-are-not-applied-properly/8182/3 "2018-01-11T10:06:16Z")

</div>

@vincent the scheduler is deployed in the default templater, right:  
 ![image](https://us1.discourse-cdn.com/flex022/uploads/suse/original/2X/e/ed5f02cb339c5d5e8a4f212578b1b27d6b5800c2.png)

I did remove `io.rancher.container.system` since rancher was deploying my lb on the host, completly taking that part of the infra down, since the IP changes…

Maybe i got confused, but how would i mark a host using a host label that way, that only catalog items are deployed, which have a specific label? I am fine with not taking the io.rancher namespace of course, so what i would use in my case would be

`de.kontextwork.concourse.affinity=workers`

I would apply this as a required label on that one host, right? This would also make this host not receive any other services which do not have this label? Because that is what happened when i upgraded to rancheros 1.1.3 - due to the restart, all the services have been rescheduled and actually scheduled on the host with that label - instead of staying on the other host without a required label.

Where did i get confused?

---

<div class="post-metadata">

**Author:** ![eesprit](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/eesprit/32/4051_2.png) [@eesprit](https://forums.suse.com/u/eesprit)\
**Post date:** [February 14, 2018, 4:24pm UTC](https://forums.suse.com/t/host-required-labels-are-not-applied-properly/8182/4 "2018-02-14T16:24:34Z")

</div>

Just got the same problem here.  
Rancher version is 1.6.13.

Historically we had three rancher hosts. Today I added a fourth one with a required label (datacenter=test).  
It also have the label “io.rancher.system” as a required label (and by the way, during tests I also did not put that label on it but have seen all infrasructure containers gets scheduled on it without problems…).

Then, I had to reboot one of the 3 host (which not have any required labels), and surprisingly I found out that some services which were on this host have been scheduled on the new one, even if they don’t have the required label (datacenter=test) on them, and they are not system (or infrastructure wide) services.

On the other hand, they have one paticular scheduling policy : we ask that there is 3 instances running (scale), and we also ask that an instance is not located on a host which already have a similar service :  
io.rancher.scheduler.affinity:container\_label\_ne: io.rancher.stack\_service.name=blabla/blabla

Don’t know if that might cause the bypassing of the required label rule on the fourth host.

Another thing that might have helped this problem to happen, is that the scheduling service was running on the node I had to reboot, and it then get rescheduled on the fourth node to.

I don’t know if the rescheduling of other services happened during the time the scheduling service was down (is that possible ?).

Anyway, that’s sound very bad to me, but perhaps I am missing something very simple.  
Or perhaps it’s a bug ?

I’ll try to do some more tests and will report on this.

@EugenMayer for the load-balancer being scheduled on the required label hosts even without the label, the solution is to add a scheduling rule on the load-balancer service to ask it to NOT be scheduled on a host with a specific label (and of course, to add that label on all hosts that you don’t want your services to go to) 😉

---

<div class="post-metadata">

**Author:** ![eesprit](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/eesprit/32/4051_2.png) [@eesprit](https://forums.suse.com/u/eesprit)\
**Post date:** [February 15, 2018, 9:06am UTC](https://forums.suse.com/t/host-required-labels-are-not-applied-properly/8182/5 "2018-02-15T09:06:43Z")

</div>

Ok, so I ran some more tests.

I created a new service, with a schedule rule to ask them to not be scheduled on a host where there is already an instance of this service.  
I scaled to 3 containers, they all went on the first 3 nodes (those without any required labels rules).  
I then scaled to 4, and got an error that it cannot find an host to run the new instance : it works as expected.  
Then I temporarily stopped the scheduler service, and tried to scale to 4 my test service again : the container is scheduled on the fourth node which defined required labels.

So that explains what happened to me yesterday : during the time that the scheduler was rescheduled on a host, some other instances have been scheduled on my fourth host, even if they don’t have the required labels.

For me it’s a big disappointment, because that means that in some special cases (losing the host running the instance of the scheduler), the “required labels” policy is not enforced, which could have bad consequences depending how we were relying on it.

Perhaps I am missing something, and it is possible to have the scheduler in some sort of high-availability setup (didn’t find docs about that), but if not, it makes the “required label” feature useless, because it could not be trusted at all time.

---

<div class="post-metadata">

**Author:** ![eesprit](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/eesprit/32/4051_2.png) [@eesprit](https://forums.suse.com/u/eesprit)\
**Post date:** [February 15, 2018, 1:41pm UTC](https://forums.suse.com/t/host-required-labels-are-not-applied-properly/8182/6 "2018-02-15T13:41:27Z")

</div>

This issue seems to be known : [https://github.com/rancher/rancher/issues/7951](https://github.com/rancher/rancher/issues/7951)  
It was rescheduled for an 1.7 milestone, but with 2.0 coming, I doubt it will ever get fixed ☹
