# Issue with network agents

**URL:** <https://forums.suse.com/t/issue-with-network-agents/261>\
**Category:** Rancher 1.x\
**Created:** [August 18, 2015, 8:58pm UTC](https://forums.suse.com/t/issue-with-network-agents/261 "2015-08-18T20:58:05Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![jgmartin](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/jgmartin/32/115_2.png) [@jgmartin](https://forums.suse.com/u/jgmartin)\
**Post date:** [August 18, 2015, 8:58pm UTC](https://forums.suse.com/t/issue-with-network-agents/261/1 "2015-08-18T20:58:05Z")

</div>

Hello all. We’ve run into an issue where a container was pushed that accidentally exposed host ports that another container already had exposed. After doing this, other stacks deployed after were failing to connect to one another (just looping in ‘networking…’ forever and failing to resolved linked host names). We tried restarting the networking agent on the hosts, but one of them is stuck trying to stop (just says ‘stopping’ forever). Deleting stacks also appears to be locked up, as the containers just spin forever on the hosts infinitely looping between stopping and networking. Any ideas how to recover from this?

---

<div class="post-metadata">

**Author:** ![tobowers](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/tobowers/32/22_2.png) [@tobowers](https://forums.suse.com/u/tobowers)\
**Post date:** [August 19, 2015, 12:09am UTC](https://forums.suse.com/t/issue-with-network-agents/261/2 "2015-08-19T00:09:45Z")

</div>

To add a little more information… we’re seeing a lot of this in the logs:

> <https://gist.github.com/tobowers/f506b96d64d2d6eabfe8>

We’re currently on: v0.31.0 with agents .

I restarted docker on the one host where networking was spinning and removed all stopped containers then put the agent back on the box. The UI still shows all the old containers on that host.

Now seeing the following in the logs:

> 2015-08-19 00:07:37,332 ERROR [:] [orService-30615] [.p.c.v.i.ConfigItemStatusManagerImpl] Failed update [configscripts, services, agent-instance-scripts, monit, agent-instance-startup, node-services, ipsec, healthcheck, ipsec-hosts, hosts, iptables, healthcheck] on [agent:3], exit code [1] output [nsenter: failed to execute /var/lib/cattle/events/config.update: No such file or directory

---

<div class="post-metadata">

**Author:** ![tobowers](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/tobowers/32/22_2.png) [@tobowers](https://forums.suse.com/u/tobowers)\
**Post date:** [August 19, 2015, 2:37pm UTC](https://forums.suse.com/t/issue-with-network-agents/261/3 "2015-08-19T14:37:45Z")

</div>

We have a deployment going on that’s taking a while… one of the hosts keeps saying ‘reconnecting’ and the python agent is using a ton of CPU:

root 10895 46.6 0.7 71996 27380 ? R 14:36 0:01 python /var/lib/cattle/pyagent/main.py

85 -96% of CPU

---

<div class="post-metadata">

**Author:** ![tobowers](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/tobowers/32/22_2.png) [@tobowers](https://forums.suse.com/u/tobowers)\
**Post date:** [August 19, 2015, 3:26pm UTC](https://forums.suse.com/t/issue-with-network-agents/261/4 "2015-08-19T15:26:13Z")

</div>

new information is that docker _itself_ appears to be hung on our agent. `docker ps` just hangs and there is nothing in the journalctl logs for docker int he last 2 minutes.

---

<div class="post-metadata">

**Author:** ![denise](https://avatars.discourse-cdn.com/v4/letter/d/82dd89/32.png) [@denise](https://forums.suse.com/u/denise)\
**Post date:** [August 19, 2015, 4:33pm UTC](https://forums.suse.com/t/issue-with-network-agents/261/5 "2015-08-19T16:33:50Z")

</div>

Would you be able to restart docker to see if that helps? When docker hangs, we won’t be able to stop/start containers.

Can you provide the rancher agent logs as well?

---

<div class="post-metadata">

**Author:** ![tobowers](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/tobowers/32/22_2.png) [@tobowers](https://forums.suse.com/u/tobowers)\
**Post date:** [August 19, 2015, 5:08pm UTC](https://forums.suse.com/t/issue-with-network-agents/261/6 "2015-08-19T17:08:45Z")

</div>

restarting docker fixed the problem (restarted docker, then started a new agent).

This doesn’t seem like a real option for production though as it means taking down a whole host.

Our current working theory was that a docker-compose with a “build” caused the agent to consume tons of resources and brought the agent down… but we don’t have evidence.

Those agent logs are gone now unfortunately I think?
