Michael Ciccarelli
2016-Oct-06 00:56 UTC
[Gluster-users] gluster and LIO, fairly basic setup, having major issues
So I have a fairly basic setup using glusterfs between 2 nodes. The nodes have 10 gig connections and the bricks reside on SSD LVM LUNs: Brick1: media1-be:/gluster/brick1/gluster_volume_0 Brick2: media2-be:/gluster/brick1/gluster_volume_0 On this volume I have a LIO iscsi target with 1 fileio backstore that's being shared out to vmware ESXi hosts. The volume is around 900 gig and the fileio store is around 850g: -rw-r--r-- 1 root root 912680550400 Oct 5 20:47 iscsi.disk.3 I set the WWN to be the same so the ESXi hosts see the nodes as 2 paths to the same target. I believe this is what I want. The issues I'm seeing is that while the IO wait is low I'm seeing high CPU usage with only 3 VMs running on only 1 of the ESX servers: this is media2-be: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1474 root 20 0 1396620 37912 5980 S 135.0 0.1 157:01.84 glusterfsd 1469 root 20 0 747996 13724 5424 S 2.0 0.0 1:10.59 glusterfs And this morning it seemed like I had to restart the LIO service on media1-be as the VMware was seeing time-out issues. I'm seeing issues like this on the VMware ESX servers: 2016-10-06T00:51:41.100Z cpu0:32785)WARNING: ScsiDeviceIO: 1223: Device naa.600140501ce79002e724ebdb66a6756d performance has deteriorated. I/O latency increased from average value of 33420 microseconds to 732696 microseconds. Are there any special settings I need to have gluster+LIO+vmware to work? Has anyone gotten this to work fairly well that it is stable? What am I missing? thanks, Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20161005/113e35e9/attachment.html>
Vijay Bellur
2016-Oct-06 16:22 UTC
[Gluster-users] gluster and LIO, fairly basic setup, having major issues
Hi Mike, Can you please share your gluster volume configuration? Also do you notice anything in client logs on the node where fileio backstore is configured? Thanks, Vijay On Wed, Oct 5, 2016 at 8:56 PM, Michael Ciccarelli <mikecicc01 at gmail.com> wrote:> So I have a fairly basic setup using glusterfs between 2 nodes. The nodes > have 10 gig connections and the bricks reside on SSD LVM LUNs: > > Brick1: media1-be:/gluster/brick1/gluster_volume_0 > Brick2: media2-be:/gluster/brick1/gluster_volume_0 > > > On this volume I have a LIO iscsi target with 1 fileio backstore that's > being shared out to vmware ESXi hosts. The volume is around 900 gig and the > fileio store is around 850g: > > -rw-r--r-- 1 root root 912680550400 Oct 5 20:47 iscsi.disk.3 > > I set the WWN to be the same so the ESXi hosts see the nodes as 2 paths to > the same target. I believe this is what I want. The issues I'm seeing is > that while the IO wait is low I'm seeing high CPU usage with only 3 VMs > running on only 1 of the ESX servers: > > this is media2-be: > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 1474 root 20 0 1396620 37912 5980 S 135.0 0.1 157:01.84 > glusterfsd > 1469 root 20 0 747996 13724 5424 S 2.0 0.0 1:10.59 > glusterfs > > And this morning it seemed like I had to restart the LIO service on > media1-be as the VMware was seeing time-out issues. I'm seeing issues like > this on the VMware ESX servers: > > 2016-10-06T00:51:41.100Z cpu0:32785)WARNING: ScsiDeviceIO: 1223: Device > naa.600140501ce79002e724ebdb66a6756d performance has deteriorated. I/O > latency increased from average value of 33420 microseconds to 732696 > microseconds. > > Are there any special settings I need to have gluster+LIO+vmware to work? > Has anyone gotten this to work fairly well that it is stable? What am I > missing? > > thanks, > Mike > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users