# Glusterfs and elasticsearch **URL:** \ **Category:** Convoy\ **Created:** [April 3, 2016, 2:23pm UTC](https://forums.suse.com/t/glusterfs-and-elasticsearch/2293 "2016-04-03T14:23:53Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![bonovoxly](https://avatars.discourse-cdn.com/v4/letter/b/cdc98d/32.png) [@bonovoxly](https://forums.suse.com/u/bonovoxly)\ **Post date:** [April 3, 2016, 2:23pm UTC](https://forums.suse.com/t/glusterfs-and-elasticsearch/2293/1 "2016-04-03T14:23:53Z")
I’m seeing issues with running elasticsearch on a glusterfs volume. I’ve configured Convoy with Glusterfs and the volume is mounted. All seems well. When elasticsearch starts up, I see this: `4/3/2016 8:53:55 AM[2016-04-03 12:53:55,083][WARN][cluster.action.shard] [Nicole St. Croix] [.kibana][0] received shard failed for target shard [[.kibana][0], node[MlquNusiR2O-9Lc5x2dLeQ], [P], v[1], s[INITIALIZING], a[id=j_vQF1yPRMWPQR56c3Th_w], unassigned_info[[reason=INDEX_CREATED], at[2016-04-03T12:53:50.855Z]]], indexUUID [_4mgkEHzRxawb_2dE3vihA], message [failed recovery], failure [IndexShardRecoveryException[failed recovery]; nested: AlreadyClosedException[Underlying file changed by an external force at 2016-04-03T12:53:54.102783Z, (lock=NativeFSLock(path=/usr/share/elasticsearch/data/elasticsearch/nodes/0/indices/.kibana/0/index/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],ctime=2016-04-03T12:53:54.102783Z))]; ] 4/3/2016 8:53:55 AM[.kibana][[.kibana][0]] IndexShardRecoveryException[failed recovery]; nested: AlreadyClosedException[Underlying file changed by an external force at 2016-04-03T12:53:54.102783Z, (lock=NativeFSLock(path=/usr/share/elasticsearch/data/elasticsearch/nodes/0/indices/.kibana/0/index/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],ctime=2016-04-03T12:53:54.102783Z))]; 4/3/2016 8:53:55 AM at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:179)` I’ve seen recommendations for enabling “cluster.consistent-metadata” for a gluster volume, however, it seems that isn’t possible for the current shipped version of glusterfs with Rancher. Any thoughts for this? I’m guessing I’ll just have to forgo this for sidekicks. ---
**Author:** ![denise](https://avatars.discourse-cdn.com/v4/letter/d/82dd89/32.png) [@denise](https://forums.suse.com/u/denise)\ **Post date:** [April 9, 2016, 3:39am UTC](https://forums.suse.com/t/glusterfs-and-elasticsearch/2293/2 "2016-04-09T03:39:17Z")
Are you launching your own version of Elastic search or how are you trying to connect it to glusterfs? Can you provide the docker-compose.yml of how you are trying to link elastic search? ---
**Author:** ![bonovoxly](https://avatars.discourse-cdn.com/v4/letter/b/cdc98d/32.png) [@bonovoxly](https://forums.suse.com/u/bonovoxly)\ **Post date:** [April 9, 2016, 1:04pm UTC](https://forums.suse.com/t/glusterfs-and-elasticsearch/2293/3 "2016-04-09T13:04:09Z")
Sure. I’m actually using it through Convoy, a volume mount. Here’s part of the docker-compose: `elasticsearch: ports: - 9200:9200/tcp - 9300:9300/tcp labels: io.rancher.scheduler.affinity:container\_label\_soft\_ne: io.rancher.stack.name=$${stack\_name} command: - elasticsearch - -Des.network.host=0.0.0.0 image: elasticsearch:latest volume-driver: convoy-gluster volumes: - elasticsearch-data:/usr/share/elasticsearch/data` I have other volume mounts to the Convoy storage mounts that work great. Logs and configs for example. But it seems with Elasticsearch, there’s something with how often it writes/reads that Convoy/GlusterFS seems to interfere with. ---
**Author:** ![rsmith](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/rsmith/32/4462_2.png) [@rsmith](https://forums.suse.com/u/rsmith)\ **Post date:** [May 9, 2016, 2:57pm UTC](https://forums.suse.com/t/glusterfs-and-elasticsearch/2293/4 "2016-05-09T14:57:20Z")
I am seeing the same issue as you are. Running the ELK stack without using a shared volume through Convoy-Gluster works correctly, but with it I am seeing the same issues. Have you found a solution to this @bonovoxly? ---
**Author:** ![davidcunningham](https://avatars.discourse-cdn.com/v4/letter/d/97f17d/32.png) [@davidcunningham](https://forums.suse.com/u/davidcunningham)\ **Post date:** [July 16, 2016, 1:00am UTC](https://forums.suse.com/t/glusterfs-and-elasticsearch/2293/5 "2016-07-16T01:00:27Z")
Anyone get this working yet? ---
**Author:** ![bonovoxly](https://avatars.discourse-cdn.com/v4/letter/b/cdc98d/32.png) [@bonovoxly](https://forums.suse.com/u/bonovoxly)\ **Post date:** [July 18, 2016, 4:44pm UTC](https://forums.suse.com/t/glusterfs-and-elasticsearch/2293/6 "2016-07-18T16:44:31Z")
@rsmith and @davidcunningham I’ve been working on a few other things and haven’t revisited this. 😞 ---
**Author:** ![hiscal](https://avatars.discourse-cdn.com/v4/letter/h/9de0a6/32.png) [@hiscal](https://forums.suse.com/u/hiscal)\ **Post date:** [May 2, 2017, 1:25am UTC](https://forums.suse.com/t/glusterfs-and-elasticsearch/2293/7 "2017-05-02T01:25:14Z")
@bonovoxly The same issue with glusterfs 3.10.1, haven’t found any solutions. ---
**Author:** ![adawolfs](https://sea2.discourse-cdn.com/flex022/user_avatar/forums.suse.com/adawolfs/32/4821_2.png) [@adawolfs](https://forums.suse.com/u/adawolfs)\ **Post date:** [February 7, 2019, 7:43pm UTC](https://forums.suse.com/t/glusterfs-and-elasticsearch/2293/8 "2019-02-07T19:43:28Z")
This happens because the ES compares ctime to check the lock file unchanged: > In GlusterFS, it returns ctime from one of the multiple backend bricks so the ctime varies: [https://bugzilla.redhat.com/show\_bug.cgi?id=1318493](https://bugzilla.redhat.com/show_bug.cgi?id=1318493) As a result, ES believes the file is changed by someone else.