# Getting errors after 1.0.1 upgrade

**URL:** <https://forums.suse.com/t/getting-errors-after-1-0-1-upgrade/2801>\
**Category:** Rancher 1.x\
**Created:** [May 15, 2016, 12:30pm UTC](https://forums.suse.com/t/getting-errors-after-1-0-1-upgrade/2801 "2016-05-15T12:30:02Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![kiboro](https://avatars.discourse-cdn.com/v4/letter/k/7993a0/32.png) [@kiboro](https://forums.suse.com/u/kiboro)\
**Post date:** [May 15, 2016, 12:30pm UTC](https://forums.suse.com/t/getting-errors-after-1-0-1-upgrade/2801/1 "2016-05-15T12:30:02Z")

</div>

Getting lots of errors after 0.59.1 to 1.0.1 upgrade. Everything seems to be working but these worry me. Can someone please explain what they mean?

```auto
2016-05-15 12:28:10,575 ERROR [772d9fe9-4399-4e7d-b445-f0fb4d288c64:5434105] [instance:127327] [instance.start->(InstanceStart)->instance.allocate->(InstanceAllocate)] [] [cutorService-37] [c.p.a.e.i.AllocatorEventListenerImpl] No allocator handled [EventVO [id=3f0c82e3-9cb9-493e-85b6-4b37bdb1d02c, name=instance.allocate, previousNames=null, replyTo=reply.2950512530058910922, resourceId=127327, resourceType=instance, publisher=null, transitioning=null, transitioningMessage=null, transitioningInternalMessage=null, previousIds=null, data={}, time=Sun May 15 12:28:10 UTC 2016, listenerKey=null, transitioningProgress=null]]
2016-05-15 12:28:10,579 ERROR [772d9fe9-4399-4e7d-b445-f0fb4d288c64:5434105] [instance:127327] [instance.start->(InstanceStart)->instance.allocate] [] [cutorService-18] [c.p.e.p.i.DefaultProcessInstanceImpl] Unknown exception io.cattle.platform.eventing.exception.EventExecutionException: Scheduling failed: No candidates available
        at io.cattle.platform.eventing.exception.EventExecutionException.fromEvent(EventExecutionException.java:53) ~[cattle-framework-eventing-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.eventing.impl.AbstractEventService.callSync(AbstractEventService.java:258) ~[cattle-framework-eventing-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.process.common.handler.EventBasedProcessHandler.handle(EventBasedProcessHandler.java:109) ~[cattle-iaas-logic-common-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.process.instance.InstanceAllocate.handle(InstanceAllocate.java:48) ~[cattle-iaas-logic-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandler(DefaultProcessInstanceImpl.java:446) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:393) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:387) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.idempotent.Idempotent.execute(Idempotent.java:42) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandlers(DefaultProcessInstanceImpl.java:387) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runLogic(DefaultProcessInstanceImpl.java:493) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runWithProcessLock(DefaultProcessInstanceImpl.java:320) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$2.doWithLockNoResult(DefaultProcessInstanceImpl.java:260) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:7) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:3) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.acquireLockAndRun(DefaultProcessInstanceImpl.java:257) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runDelegateLoop(DefaultProcessInstanceImpl.java:185) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.executeWithProcessInstanceLock(DefaultProcessInstanceImpl.java:158) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:108) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:105) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.execute(DefaultProcessInstanceImpl.java:105) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.process.common.handler.AbstractObjectProcessLogic.execute(AbstractObjectProcessLogic.java:131) [cattle-iaas-logic-common-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.process.instance.InstanceStart.allocate(InstanceStart.java:217) [cattle-iaas-logic-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.process.instance.InstanceStart.handle(InstanceStart.java:75) [cattle-iaas-logic-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandler(DefaultProcessInstanceImpl.java:446) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:393) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:387) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.idempotent.Idempotent.execute(Idempotent.java:42) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandlers(DefaultProcessInstanceImpl.java:387) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runLogic(DefaultProcessInstanceImpl.java:493) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runWithProcessLock(DefaultProcessInstanceImpl.java:320) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$2.doWithLockNoResult(DefaultProcessInstanceImpl.java:260) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:7) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:3) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.acquireLockAndRun(DefaultProcessInstanceImpl.java:257) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runDelegateLoop(DefaultProcessInstanceImpl.java:185) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.executeWithProcessInstanceLock(DefaultProcessInstanceImpl.java:158) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:108) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:105) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.execute(DefaultProcessInstanceImpl.java:105) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.eventing.impl.ProcessEventListenerImpl.processExecute(ProcessEventListenerImpl.java:74) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.eventing.impl.ProcessEventListenerImpl.processExecute(ProcessEventListenerImpl.java:56) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at sun.reflect.GeneratedMethodAccessor461.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_95]
        at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_95]
        at io.cattle.platform.eventing.annotation.MethodInvokingListener$1.doWithLockNoResult(MethodInvokingListener.java:76) [cattle-framework-eventing-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:7) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:3) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.eventing.annotation.MethodInvokingListener.onEvent(MethodInvokingListener.java:72) [cattle-framework-eventing-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.eventing.impl.AbstractThreadPoolingEventService$2.doRun(AbstractThreadPoolingEventService.java:135) [cattle-framework-eventing-0.5.0-SNAPSHOT.jar:na]
        at org.apache.cloudstack.managed.context.NoExceptionRunnable.runInContext(NoExceptionRunnable.java:15) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:108) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_95]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_95]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_95]

2016-05-15 12:28:10,579 ERROR [772d9fe9-4399-4e7d-b445-f0fb4d288c64:5434105] [instance:127327] [instance.start->(InstanceStart)] [] [cutorService-18] [i.c.p.process.instance.InstanceStart] Failed to Scheduling for instance [127327]

```

Further info. For that container name I have the following records in instance:

```auto
mysql> select count(1),state from instance where name = 'infrastructure_keycloak_1' group by state;
+----------+---------+
| count(1) | state |
+----------+---------+
| 1 | error |
| 47726 | purged |
| 1 | removed |
| 3 | running |
+----------+---------+

```

Can I safely remove all the purged ones? The error and removed are not present and shouldn’t be. I should have one each in Test,UAT and Live environments

---

<div class="post-metadata">

**Author:** ![denise](https://avatars.discourse-cdn.com/v4/letter/d/82dd89/32.png) [@denise](https://forums.suse.com/u/denise)\
**Post date:** [May 15, 2016, 10:16pm UTC](https://forums.suse.com/t/getting-errors-after-1-0-1-upgrade/2801/2 "2016-05-15T22:16:28Z")

</div>

> [@kiboro](#):
>
> Failed to Scheduling for instance [127328]

There is a container for a service that is trying to launch, but is unable to. Do you have any services that are unhealthy?

Are you able to poke around the DB to try and find which instance has the ID `12738`?

---

<div class="post-metadata">

**Author:** ![kiboro](https://avatars.discourse-cdn.com/v4/letter/k/7993a0/32.png) [@kiboro](https://forums.suse.com/u/kiboro)\
**Post date:** [May 19, 2016, 1:56pm UTC](https://forums.suse.com/t/getting-errors-after-1-0-1-upgrade/2801/3 "2016-05-19T13:56:21Z")

</div>

Sorry Denise, was away at an off-site meeting for days. It appears that that instance has a state of purged so shouldn’t be a target to start anyway.

```auto
| 127328 | infrastructure_keycloak-db_1 | 60 | container | 835124ec-96ec-4669-b9ca-cdc082dce918 | NULL | purged | 2016-05-15 12:28:07 | 2016-05-15 12:29:51 | 2016-05-15 12:30:51 | {"fields":{"serviceIndex":"1","imageUuid":"docker:mysql/mysql-server:5.6","allocatedIpAddress":null,"publishAllPorts":false,"startOnCreate":true,"labels":{"io.rancher.project.name":"infrastructure","io.rancher.service.deployment.unit":"0ab14901-0fbc-4296-89ac-b3de47dfcbf7","io.rancher.service.launch.config":"io.rancher.service.primary.launch.config","io.rancher.project_service.name":"infrastructure/keycloak-db","io.rancher.stack.name":"infrastructure","io.rancher.stack_service.name":"infrastructure/keycloak-db","io.rancher.service.hash":"eacc1d7b761cfc59f35c7fcf1bb558a8962d7ab0"},"networkMode":"managed","stdinOpen":true,"dnsSearch":["infrastructure.rancher.internal","keycloak-db.infrastructure.rancher.internal","rancher.internal"],"vcpu":1,"environment":{"MYSQL_DATABASE":"keycloak","MYSQL_PASSWORD":"xxxxxxx","MYSQL_ROOT_PASSWORD":"xxxxxxx","MYSQL_USER":"keycloak"},"readOnly":false,"tty":true,"privileged":false,"logConfig":{},"dns":["169.254.169.250"],"networkIds":[30],"dataVolumeMounts":{},"dataVolumes":[],"transitioningProgress":0}} | inactive | NULL | NULL | 127284 | NULL | NULL | 1 | stop | NULL | NULL | NULL | pDameugUzE7sRO6zuSuz47PGVGpJSKqZPmPJM2TN35mAW9wsQv3L8MoQjhJ0B6KcX5q7B3SERrHkUv77DcQ | NULL | NULL | NULL | NULL | | NULL | NULL | 0 | 45476 | 0ab14901-0fbc-4296-89ac-b3de47dfcbf7 | 0 | NULL | 41 | 169.254.169.250 | infrastructure.rancher.internal,keycloak-db.infrastructure.rancher.internal,rancher.internal |

```

---

<div class="post-metadata">

**Author:** ![kiboro](https://avatars.discourse-cdn.com/v4/letter/k/7993a0/32.png) [@kiboro](https://forums.suse.com/u/kiboro)\
**Post date:** [May 19, 2016, 1:59pm UTC](https://forums.suse.com/t/getting-errors-after-1-0-1-upgrade/2801/4 "2016-05-19T13:59:36Z")

</div>

I’m also getting similar things from volumes which are purged

```auto
2016-05-19 13:48:19,048 ERROR [:] [] [] [] [erviceReplay-23] [i.c.p.e.e.i.ProcessEventListenerImpl] Unknown exception running process [volume.activate:3516179] on [37235] io.cattle.platform.eventing.exception.AgentRemovedException: Agent is removed

```

Are these from a removed host?

---

<div class="post-metadata">

**Author:** ![denise](https://avatars.discourse-cdn.com/v4/letter/d/82dd89/32.png) [@denise](https://forums.suse.com/u/denise)\
**Post date:** [May 20, 2016, 5:10am UTC](https://forums.suse.com/t/getting-errors-after-1-0-1-upgrade/2801/5 "2016-05-20T05:10:21Z")

</div>

@kiboro, Yes those exceptions are from a removed host that should be resolved if you just restart your Rancher server.

We have a couple of issues in Github for the exception.

> <https://github.com/rancher/rancher/issues/4682>

  

> <https://github.com/rancher/rancher/issues/4686>

---

<div class="post-metadata">

**Author:** ![kiboro](https://avatars.discourse-cdn.com/v4/letter/k/7993a0/32.png) [@kiboro](https://forums.suse.com/u/kiboro)\
**Post date:** [May 20, 2016, 9:50am UTC](https://forums.suse.com/t/getting-errors-after-1-0-1-upgrade/2801/6 "2016-05-20T09:50:53Z")

</div>

But this started happening after upgrading to 1.0.1 so by definition a restart