# Containers stuck at "scheduling"

**URL:** <https://forums.suse.com/t/containers-stuck-at-scheduling/2441>\
**Category:** Rancher 1.x\
**Created:** [April 14, 2016, 2:13am UTC](https://forums.suse.com/t/containers-stuck-at-scheduling/2441 "2016-04-14T02:13:07Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![jlk](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/jlk/32/1173_2.png) [@jlk](https://forums.suse.com/u/jlk)\
**Post date:** [April 14, 2016, 2:13am UTC](https://forums.suse.com/t/containers-stuck-at-scheduling/2441/1 "2016-04-14T02:13:07Z")

</div>

Hey folks. The last few containers I try to start from rancher-compose seem to be getting stuck at “scheduling.” If I drill in on a service from a stack, I see a status for the service of “Activating (Waiting for [instance:clair\_clair\_1]. Instance status: Scheduling)”

Looking at rancher-server logs, I see the following stack trace being repeated:

```
2016-04-14 01:44:43,920 ERROR [:] [] [] [] [ServiceReplay-2] [i.c.p.e.e.i.ProcessEventListenerImpl] Unknown exception running process [instance.start:15109] on [230] java.lang.IllegalStateException: Attempt to cancel when process is still transitioning
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runDelegateLoop(DefaultProcessInstanceImpl.java:191) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.executeWithProcessInstanceLock(DefaultProcessInstanceImpl.java:158) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:108) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:105) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.execute(DefaultProcessInstanceImpl.java:105) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.process.common.handler.AbstractObjectProcessLogic.execute(AbstractObjectProcessLogic.java:131) ~[cattle-iaas-logic-common-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.process.instance.InstanceStart.allocate(InstanceStart.java:217) ~[cattle-iaas-logic-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.process.instance.InstanceStart.handle(InstanceStart.java:75) ~[cattle-iaas-logic-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandler(DefaultProcessInstanceImpl.java:446) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:393) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:387) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.idempotent.Idempotent.execute(Idempotent.java:42) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandlers(DefaultProcessInstanceImpl.java:387) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runLogic(DefaultProcessInstanceImpl.java:493) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runWithProcessLock(DefaultProcessInstanceImpl.java:320) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$2.doWithLockNoResult(DefaultProcessInstanceImpl.java:260) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:7) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:3) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.acquireLockAndRun(DefaultProcessInstanceImpl.java:257) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runDelegateLoop(DefaultProcessInstanceImpl.java:185) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.executeWithProcessInstanceLock(DefaultProcessInstanceImpl.java:158) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:108) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:105) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.execute(DefaultProcessInstanceImpl.java:105) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.eventing.impl.ProcessEventListenerImpl.processExecute(ProcessEventListenerImpl.java:69) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at io.cattle.platform.engine.server.impl.ProcessInstanceParallelDispatcher$1.runInContext(ProcessInstanceParallelDispatcher.java:27) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:108) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_95]
        at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_95]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_95]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_95]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_95]

```

Looking under admin/processes, I see instance.create tasks failing with “UNKNOWN EXCEPTION”

Anybody seen similar? I haven’t dug into the rancher source yet, but guessing a job got stuck somewhere and it’s screwing up anything that comes after it…

Happy to add a docker-compose spec or whatever, but this doesn’t seem to be tied to the actual container definition itself

---

<div class="post-metadata">

**Author:** ![jlk](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/jlk/32/1173_2.png) [@jlk](https://forums.suse.com/u/jlk)\
**Post date:** [April 14, 2016, 4:54am UTC](https://forums.suse.com/t/containers-stuck-at-scheduling/2441/2 "2016-04-14T04:54:58Z")

</div>

Update…

Backed up the db and did a little poking around. Good amount of carnage in process\_instance table. Tried updating records that had null end\_time values, then records that had null results…ended up just truncating the process\_instance table. That seemed to help a little. Was able to launch wordpress from the catalog and that came up. So went back and tried firing up my service with rancher-compose, service now goes to “Activating (Reconciling)” and then the same Exception fires.

I’ve stared at this enough for one night - I suspect might be issues in the instance table, will dig further in the am.

---

<div class="post-metadata">

**Author:** ![cjellick](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/cjellick/32/2876_2.png) [@cjellick](https://forums.suse.com/u/cjellick)\
**Post date:** [April 14, 2016, 3:49pm UTC](https://forums.suse.com/t/containers-stuck-at-scheduling/2441/3 "2016-04-14T15:49:42Z")

</div>

@jlk can you share some more details about your environment? Rancher version, docker version, number of hosts, and the compose you are trying to run.

Also, can you share a bigger chunk of your logs from before and after the exception? You can put those in a gist if you like instead of pasting directly into this thread.

---

<div class="post-metadata">

**Author:** ![jlk](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/jlk/32/1173_2.png) [@jlk](https://forums.suse.com/u/jlk)\
**Post date:** [April 14, 2016, 4:56pm UTC](https://forums.suse.com/t/containers-stuck-at-scheduling/2441/4 "2016-04-14T16:56:15Z")

</div>

Sure

Rancher v1.0  
On the Rancher server, docker 1.10.3 running on Ubuntu 14.04  
ec2 hosts are provisioned by Rancher - 1.10.3 on Ubuntu 15.10, m4.large instances. 2 of those in this case. Plenty of free cpu/mem/disk.

Gist of a few thousand lines at [https://gist.github.com/jlk/3ca9206a6391f9abf895631d252c6c09](https://gist.github.com/jlk/3ca9206a6391f9abf895631d252c6c09). Not trying to overload, but it mostly looks repetitive…

For the compose file, I’m down to trying something fairly simplistic:

```
  postgres:
    image: postgres:latest
    volumes:
      - postgres:/var/lib/postgresql/data
    volume_driver: convoy-nfs
    environment:
      POSTGRES_PASSWORD: password
```

---

<div class="post-metadata">

**Author:** ![jlk](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/jlk/32/1173_2.png) [@jlk](https://forums.suse.com/u/jlk)\
**Post date:** [April 15, 2016, 12:21am UTC](https://forums.suse.com/t/containers-stuck-at-scheduling/2441/5 "2016-04-15T00:21:56Z")

</div>

I nuked and installed v1.0.1…if I see this again, will pick it back up…

---

<div class="post-metadata">

**Author:** ![Hao\_Wan](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/hao_wan/32/1789_2.png) [@Hao\_Wan](https://forums.suse.com/u/Hao_Wan)\
**Post date:** [October 18, 2016, 10:37am UTC](https://forums.suse.com/t/containers-stuck-at-scheduling/2441/6 "2016-10-18T10:37:58Z")

</div>

@jlk, have you resolved the problem? I met the same problem.

after updating the instance in stack, the stack and service always being activating.  
Stack(Environment): activating.  
Serivce: activating  
instance: running.  
they have inconsistent status.

---

<div class="post-metadata">

**Author:** ![jlk](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/jlk/32/1173_2.png) [@jlk](https://forums.suse.com/u/jlk)\
**Post date:** [October 18, 2016, 5:55pm UTC](https://forums.suse.com/t/containers-stuck-at-scheduling/2441/7 "2016-10-18T17:55:20Z")

</div>

I’m unfortunately not currently using Rancher, so can’t say I’ve addressed it.