Gluster users - Nov 2017 - [ovirt-users] slow performance with export storage on glusterfs

What about mounting over nfs instead of the fuse client. Or maybe libgfapi.
Is that available for export domains

On Fri, Nov 24, 2017 at 3:48 AM Ji?? Sl??ka <jiri.slezka at slu.cz> wrote:
> On 11/24/2017 06:41 AM, Sahina Bose wrote:
> >
> >
> > On Thu, Nov 23, 2017 at 4:56 PM, Ji?? Sl??ka <jiri.slezka at slu.cz
> > <mailto:jiri.slezka at slu.cz>> wrote:
> >
> >     Hi,
> >
> >     On 11/22/2017 07:30 PM, Nir Soffer wrote:
> >     > On Mon, Nov 20, 2017 at 5:22 PM Ji?? Sl??ka <jiri.slezka
at slu.cz
> <mailto:jiri.slezka at slu.cz>
> >     > <mailto:jiri.slezka at slu.cz <mailto:jiri.slezka at
slu.cz>>> wrote:
> >     >
> >     >     Hi,
> >     >
> >     >     I am trying realize why is exporting of vm to export
storage on
> >     >     glusterfs such slow.
> >     >
> >     >     I am using oVirt and RHV, both instalations on version
4.1.7.
> >     >
> >     >     Hosts have dedicated nics for rhevm network - 1gbps, data
> >     storage itself
> >     >     is on FC.
> >     >
> >     >     GlusterFS cluster lives separate on 4 dedicated hosts. It
has
> >     slow disks
> >     >     but I can achieve about 200-400mbit throughput in other
> >     applications (we
> >     >     are using it for "cold" data, backups mostly).
> >     >
> >     >     I am using this glusterfs cluster as backend for export
> >     storage. When I
> >     >     am exporting vm I can see only about 60-80mbit
throughput.
> >     >
> >     >     What could be the bottleneck here?
> >     >
> >     >     Could it be qemu-img utility?
> >     >
> >     >     vdsm      97739  0.3  0.0 354212 29148 ?        S<l 
15:43
>  0:06
> >     >     /usr/bin/qemu-img convert -p -t none -T none -f raw
> >     >
> >
>  
/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
> >     >     -O raw
> >     >
> >      /rhev/data-center/mnt/glusterSD/10.20.30.41:
>
_rhv__export/81094499-a392-4ea2-b081-7c6288fbb636/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
> >     >
> >     >     Any idea how to make it work faster or what throughput
should I
> >     >     expected?
> >     >
> >     >
> >     > gluster storage operations are using fuse mount - so every
write:
> >     > - travel to the kernel
> >     > - travel back to the gluster fuse helper process
> >     > - travel to all 3 replicas - replication is done on client
side
> >     > - return to kernel when all writes succeeded
> >     > - return to caller
> >     >
> >     > So gluster will never set any speed record.
> >     >
> >     > Additionally, you are copying from raw lv on FC - qemu-img
cannot
> do
> >     > anything
> >     > smart and avoid copying unused clusters. Instead if copies
> >     gigabytes of
> >     > zeros
> >     > from FC.
> >
> >     ok, it does make sense
> >
> >     > However 7.5-10 MiB/s sounds too slow.
> >     >
> >     > I would try to test with dd - how much time it takes to copy
> >     > the same image from FC to your gluster storage?
> >     >
> >     > dd
> >     >
>
if=/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
> >     > of=/rhev/data-center/mnt/glusterSD/10.20.30.41:
> _rhv__export/81094499-a392-4ea2-b081-7c6288fbb636/__test__
> >     > bs=8M oflag=direct status=progress
> >
> >     unfrotunately dd performs the same
> >
> >     1778384896 bytes (1.8 GB) copied, 198.565265 s, 9.0 MB/s
> >
> >
> >     > If dd can do this faster, please ask on qemu-discuss mailing
list:
> >     > https://lists.nongnu.org/mailman/listinfo/qemu-discuss
> >     <https://lists.nongnu.org/mailman/listinfo/qemu-discuss>
> >     >
> >     > If both give similar results, I think asking in gluster
mailing
> list
> >     > about this can help. Maybe your gluster setup can be
optimized.
> >
> >     ok, this is definitly on the gluster side. Thanks for your
guidance.
> >
> >     I will investigate the gluster side and also will try Export on
NFS
> >     share.
> >
> >
> > [Adding gluster users ml]
> >
> > Please provide "gluster volume info" output for the
rhv_export gluster
> > volume and also volume profile details (refer to earlier mail from
Shani
> > on how to run this) while performing the dd operation above.
>
> you can find all this output on https://pastebin.com/sBK01VS8
>
> as mentioned in other posts. Gluster cluster uses really slow (green)
> disks but without direct io it can achieve throughput around 400mbit/s.
>
> This storage is used mostly for backup purposes. It is not used as a vm
> storage.
>
> In my case it would be nice not to use direct io in export case but I
> understand why it might not be wise.
>
> Cheers,
>
> Jiri
>
> >
> >
> >
> >
> >     Cheers,
> >
> >     Jiri
> >
> >
> >     >
> >     > Nir
> >     >
> >     >
> >     >
> >     >     Cheers,
> >     >
> >     >     Jiri
> >     >
> >     >
> >     >     _______________________________________________
> >     >     Users mailing list
> >     >     Users at ovirt.org <mailto:Users at ovirt.org>
> >     <mailto:Users at ovirt.org <mailto:Users at
ovirt.org>>
> >     >     http://lists.ovirt.org/mailman/listinfo/users
> >     <http://lists.ovirt.org/mailman/listinfo/users>
> >     >
> >
> >
> >
> >     _______________________________________________
> >     Users mailing list
> >     Users at ovirt.org <mailto:Users at ovirt.org>
> >     http://lists.ovirt.org/mailman/listinfo/users
> >     <http://lists.ovirt.org/mailman/listinfo/users>
> >
> >
>
>
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20171128/4c2001ad/attachment.html>

Hello

If you use Gluster as FUSE mount it's always slower than you expect it to
be.
If you want to get better performance out of your oVirt/Gluster storage,
try the following:

- create a Linux VM in your oVirt environment, assign 4/8/12 virtual disks
(Virtual disks are located on your Gluster storage volume).
- Boot/configure the VM, then use LVM to create VG/LV with 4 stripes
(lvcreate -i 4) and use all 4/8/12 virtual disks as PVS.
- then install NFS server and export LV you created in previous step, use
the NFS export as export domain in oVirt/RHEV.

You should get wire speed when you use multiple stripes on Gluster storage,
FUSE mount on oVirt host will fan out requests to all 4 servers.
Gluster is very good at distributed/parallel workloads, but when you use
direct Gluster FUSE mount for Export domain you only have one data stream,
which is fragmented even more my multiple writes/reads that Gluster needs
to do to save your data on all member servers.



On Mon, Nov 27, 2017 at 8:41 PM, Donny Davis <donny at fortnebula.com>
wrote:
> What about mounting over nfs instead of the fuse client. Or maybe
> libgfapi. Is that available for export domains
>
> On Fri, Nov 24, 2017 at 3:48 AM Ji?? Sl??ka <jiri.slezka at slu.cz>
wrote:
>
>> On 11/24/2017 06:41 AM, Sahina Bose wrote:
>> >
>> >
>> > On Thu, Nov 23, 2017 at 4:56 PM, Ji?? Sl??ka <jiri.slezka at
slu.cz
>> > <mailto:jiri.slezka at slu.cz>> wrote:
>> >
>> >     Hi,
>> >
>> >     On 11/22/2017 07:30 PM, Nir Soffer wrote:
>> >     > On Mon, Nov 20, 2017 at 5:22 PM Ji?? Sl??ka
<jiri.slezka at slu.cz
>> <mailto:jiri.slezka at slu.cz>
>> >     > <mailto:jiri.slezka at slu.cz <mailto:jiri.slezka
at slu.cz>>> wrote:
>> >     >
>> >     >     Hi,
>> >     >
>> >     >     I am trying realize why is exporting of vm to export
storage
>> on
>> >     >     glusterfs such slow.
>> >     >
>> >     >     I am using oVirt and RHV, both instalations on
version 4.1.7.
>> >     >
>> >     >     Hosts have dedicated nics for rhevm network - 1gbps,
data
>> >     storage itself
>> >     >     is on FC.
>> >     >
>> >     >     GlusterFS cluster lives separate on 4 dedicated
hosts. It has
>> >     slow disks
>> >     >     but I can achieve about 200-400mbit throughput in
other
>> >     applications (we
>> >     >     are using it for "cold" data, backups
mostly).
>> >     >
>> >     >     I am using this glusterfs cluster as backend for
export
>> >     storage. When I
>> >     >     am exporting vm I can see only about 60-80mbit
throughput.
>> >     >
>> >     >     What could be the bottleneck here?
>> >     >
>> >     >     Could it be qemu-img utility?
>> >     >
>> >     >     vdsm      97739  0.3  0.0 354212 29148 ?       
S<l  15:43
>>  0:06
>> >     >     /usr/bin/qemu-img convert -p -t none -T none -f raw
>> >     >
>> >      /rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/
>> ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-
>> c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
>> >     >     -O raw
>> >     >
>> >      /rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__
>> export/81094499-a392-4ea2-b081-7c6288fbb636/images/
>>
ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
>> >     >
>> >     >     Any idea how to make it work faster or what
throughput should
>> I
>> >     >     expected?
>> >     >
>> >     >
>> >     > gluster storage operations are using fuse mount - so
every write:
>> >     > - travel to the kernel
>> >     > - travel back to the gluster fuse helper process
>> >     > - travel to all 3 replicas - replication is done on
client side
>> >     > - return to kernel when all writes succeeded
>> >     > - return to caller
>> >     >
>> >     > So gluster will never set any speed record.
>> >     >
>> >     > Additionally, you are copying from raw lv on FC -
qemu-img cannot
>> do
>> >     > anything
>> >     > smart and avoid copying unused clusters. Instead if
copies
>> >     gigabytes of
>> >     > zeros
>> >     > from FC.
>> >
>> >     ok, it does make sense
>> >
>> >     > However 7.5-10 MiB/s sounds too slow.
>> >     >
>> >     > I would try to test with dd - how much time it takes to
copy
>> >     > the same image from FC to your gluster storage?
>> >     >
>> >     > dd
>> >     >
if=/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/
>> ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-
>> c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
>> >     > of=/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__
>> export/81094499-a392-4ea2-b081-7c6288fbb636/__test__
>> >     > bs=8M oflag=direct status=progress
>> >
>> >     unfrotunately dd performs the same
>> >
>> >     1778384896 bytes (1.8 GB) copied, 198.565265 s, 9.0 MB/s
>> >
>> >
>> >     > If dd can do this faster, please ask on qemu-discuss
mailing list:
>> >     > https://lists.nongnu.org/mailman/listinfo/qemu-discuss
>> >     <https://lists.nongnu.org/mailman/listinfo/qemu-discuss>
>> >     >
>> >     > If both give similar results, I think asking in gluster
mailing
>> list
>> >     > about this can help. Maybe your gluster setup can be
optimized.
>> >
>> >     ok, this is definitly on the gluster side. Thanks for your
guidance.
>> >
>> >     I will investigate the gluster side and also will try Export
on NFS
>> >     share.
>> >
>> >
>> > [Adding gluster users ml]
>> >
>> > Please provide "gluster volume info" output for the
rhv_export gluster
>> > volume and also volume profile details (refer to earlier mail from
Shani
>> > on how to run this) while performing the dd operation above.
>>
>> you can find all this output on https://pastebin.com/sBK01VS8
>>
>> as mentioned in other posts. Gluster cluster uses really slow (green)
>> disks but without direct io it can achieve throughput around 400mbit/s.
>>
>> This storage is used mostly for backup purposes. It is not used as a vm
>> storage.
>>
>> In my case it would be nice not to use direct io in export case but I
>> understand why it might not be wise.
>>
>> Cheers,
>>
>> Jiri
>>
>> >
>> >
>> >
>> >
>> >     Cheers,
>> >
>> >     Jiri
>> >
>> >
>> >     >
>> >     > Nir
>> >     >
>> >     >
>> >     >
>> >     >     Cheers,
>> >     >
>> >     >     Jiri
>> >     >
>> >     >
>> >     >     _______________________________________________
>> >     >     Users mailing list
>> >     >     Users at ovirt.org <mailto:Users at ovirt.org>
>> >     <mailto:Users at ovirt.org <mailto:Users at
ovirt.org>>
>> >     >     http://lists.ovirt.org/mailman/listinfo/users
>> >     <http://lists.ovirt.org/mailman/listinfo/users>
>> >     >
>> >
>> >
>> >
>> >     _______________________________________________
>> >     Users mailing list
>> >     Users at ovirt.org <mailto:Users at ovirt.org>
>> >     http://lists.ovirt.org/mailman/listinfo/users
>> >     <http://lists.ovirt.org/mailman/listinfo/users>
>> >
>> >
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20171128/173dac9a/attachment.html>

Hello,
> 
> If you use Gluster as FUSE mount it's always slower than you expect it
> to be.
> If you want to get better performance out of your oVirt/Gluster storage,
> try the following:?
> 
> - create a Linux VM in your oVirt environment, assign 4/8/12 virtual
> disks (Virtual disks are located on your Gluster storage volume).
> - Boot/configure the VM, then use LVM to create VG/LV with 4 stripes
> (lvcreate -i 4) and use all 4/8/12 virtual disks as PVS.
> - then install NFS server and export LV you created in previous step,
> use the NFS export as export domain in oVirt/RHEV.
> 
> You should get wire speed when you use multiple stripes on Gluster
> storage, FUSE mount on oVirt host will fan out requests to all 4 servers.
> Gluster is very good at distributed/parallel workloads, but when you use
> direct Gluster FUSE mount for Export domain you only have one data
> stream, which is fragmented even more my multiple writes/reads that
> Gluster needs to do to save your data on all member servers.
Thanks for explanation, it is an interesting solution.

Cheers,

Jiri
> 
> 
> 
> On Mon, Nov 27, 2017 at 8:41 PM, Donny Davis <donny at fortnebula.com
> <mailto:donny at fortnebula.com>> wrote:
> 
>     What about mounting over nfs instead of the fuse client. Or maybe
>     libgfapi. Is that available for export domains
> 
>     On Fri, Nov 24, 2017 at 3:48 AM Ji?? Sl??ka <jiri.slezka at slu.cz
>     <mailto:jiri.slezka at slu.cz>> wrote:
> 
>         On 11/24/2017 06:41 AM, Sahina Bose wrote:
>         >
>         >
>         > On Thu, Nov 23, 2017 at 4:56 PM, Ji?? Sl??ka
>         <jiri.slezka at slu.cz <mailto:jiri.slezka at slu.cz>
>         > <mailto:jiri.slezka at slu.cz <mailto:jiri.slezka at
slu.cz>>> wrote:
>         >
>         >? ? ?Hi,
>         >
>         >? ? ?On 11/22/2017 07:30 PM, Nir Soffer wrote:
>         >? ? ?> On Mon, Nov 20, 2017 at 5:22 PM Ji?? Sl??ka
>         <jiri.slezka at slu.cz <mailto:jiri.slezka at slu.cz>
>         <mailto:jiri.slezka at slu.cz <mailto:jiri.slezka at
slu.cz>>
>         >? ? ?> <mailto:jiri.slezka at slu.cz
<mailto:jiri.slezka at slu.cz>
>         <mailto:jiri.slezka at slu.cz <mailto:jiri.slezka at
slu.cz>>>> wrote:
>         >? ? ?>
>         >? ? ?>? ? ?Hi,
>         >? ? ?>
>         >? ? ?>? ? ?I am trying realize why is exporting of vm to
export
>         storage on
>         >? ? ?>? ? ?glusterfs such slow.
>         >? ? ?>
>         >? ? ?>? ? ?I am using oVirt and RHV, both instalations on
>         version 4.1.7.
>         >? ? ?>
>         >? ? ?>? ? ?Hosts have dedicated nics for rhevm network -
1gbps,
>         data
>         >? ? ?storage itself
>         >? ? ?>? ? ?is on FC.
>         >? ? ?>
>         >? ? ?>? ? ?GlusterFS cluster lives separate on 4 dedicated
>         hosts. It has
>         >? ? ?slow disks
>         >? ? ?>? ? ?but I can achieve about 200-400mbit throughput in
other
>         >? ? ?applications (we
>         >? ? ?>? ? ?are using it for "cold" data, backups
mostly).
>         >? ? ?>
>         >? ? ?>? ? ?I am using this glusterfs cluster as backend for
export
>         >? ? ?storage. When I
>         >? ? ?>? ? ?am exporting vm I can see only about 60-80mbit
>         throughput.
>         >? ? ?>
>         >? ? ?>? ? ?What could be the bottleneck here?
>         >? ? ?>
>         >? ? ?>? ? ?Could it be qemu-img utility?
>         >? ? ?>
>         >? ? ?>? ? ?vdsm? ? ? 97739? 0.3? 0.0 354212 29148 ?? ? ? ?
S<l?
>         15:43? ?0:06
>         >? ? ?>? ? ?/usr/bin/qemu-img convert -p -t none -T none -f
raw
>         >? ? ?>? ?
>         >? ?
>        
??/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
>         >? ? ?>? ? ?-O raw
>         >? ? ?>? ?
>         >? ?
>        
??/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__export/81094499-a392-4ea2-b081-7c6288fbb636/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
>         >? ? ?>
>         >? ? ?>? ? ?Any idea how to make it work faster or what
>         throughput should I
>         >? ? ?>? ? ?expected?
>         >? ? ?>
>         >? ? ?>
>         >? ? ?> gluster storage operations are using fuse mount - so
>         every write:
>         >? ? ?> - travel to the kernel
>         >? ? ?> - travel back to the gluster fuse helper process
>         >? ? ?> - travel to all 3 replicas - replication is done on
>         client side
>         >? ? ?> - return to kernel when all writes succeeded
>         >? ? ?> - return to caller
>         >? ? ?>
>         >? ? ?> So gluster will never set any speed record.
>         >? ? ?>
>         >? ? ?> Additionally, you are copying from raw lv on FC -
>         qemu-img cannot do
>         >? ? ?> anything
>         >? ? ?> smart and avoid copying unused clusters. Instead if
copies
>         >? ? ?gigabytes of
>         >? ? ?> zeros
>         >? ? ?> from FC.
>         >
>         >? ? ?ok, it does make sense
>         >
>         >? ? ?> However 7.5-10 MiB/s sounds too slow.
>         >? ? ?>
>         >? ? ?> I would try to test with dd - how much time it takes
to copy
>         >? ? ?> the same image from FC to your gluster storage?
>         >? ? ?>
>         >? ? ?> dd
>         >? ? ?>
>        
if=/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
>         >? ? ?>
>        
of=/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__export/81094499-a392-4ea2-b081-7c6288fbb636/__test__
>         >? ? ?> bs=8M oflag=direct status=progress
>         >
>         >? ? ?unfrotunately dd performs the same
>         >
>         >? ? ?1778384896 bytes (1.8 GB) copied, 198.565265 s, 9.0 MB/s
>         >
>         >
>         >? ? ?> If dd can do this faster, please ask on qemu-discuss
>         mailing list:
>         >? ? ?>
https://lists.nongnu.org/mailman/listinfo/qemu-discuss
>         <https://lists.nongnu.org/mailman/listinfo/qemu-discuss>
>         >? ? ?<https://lists.nongnu.org/mailman/listinfo/qemu-discuss
>         <https://lists.nongnu.org/mailman/listinfo/qemu-discuss>>
>         >? ? ?>
>         >? ? ?> If both give similar results, I think asking in
gluster
>         mailing list
>         >? ? ?> about this can help. Maybe your gluster setup can be
>         optimized.
>         >
>         >? ? ?ok, this is definitly on the gluster side. Thanks for your
>         guidance.
>         >
>         >? ? ?I will investigate the gluster side and also will try
>         Export on NFS
>         >? ? ?share.
>         >
>         >
>         > [Adding gluster users ml]
>         >
>         > Please provide "gluster volume info" output for the
rhv_export
>         gluster
>         > volume and also volume profile details (refer to earlier mail
>         from Shani
>         > on how to run this) while performing the dd operation above.
> 
>         you can find all this output on https://pastebin.com/sBK01VS8
> 
>         as mentioned in other posts. Gluster cluster uses really slow
>         (green)
>         disks but without direct io it can achieve throughput around
>         400mbit/s.
> 
>         This storage is used mostly for backup purposes. It is not used
>         as a vm
>         storage.
> 
>         In my case it would be nice not to use direct io in export case
>         but I
>         understand why it might not be wise.
> 
>         Cheers,
> 
>         Jiri
> 
>         >
>         > ?
>         >
>         >
>         >? ? ?Cheers,
>         >
>         >? ? ?Jiri
>         >
>         >
>         >? ? ?>
>         >? ? ?> Nir
>         >? ? ?> ?
>         >? ? ?>
>         >? ? ?>
>         >? ? ?>? ? ?Cheers,
>         >? ? ?>
>         >? ? ?>? ? ?Jiri
>         >? ? ?>
>         >? ? ?>
>         >? ? ?>? ? ?_______________________________________________
>         >? ? ?>? ? ?Users mailing list
>         >? ? ?>? ? ?Users at ovirt.org <mailto:Users at
ovirt.org>
>         <mailto:Users at ovirt.org <mailto:Users at ovirt.org>>
>         >? ? ?<mailto:Users at ovirt.org <mailto:Users at
ovirt.org>
>         <mailto:Users at ovirt.org <mailto:Users at
ovirt.org>>>
>         >? ? ?>? ? ?http://lists.ovirt.org/mailman/listinfo/users
>         <http://lists.ovirt.org/mailman/listinfo/users>
>         >? ? ?<http://lists.ovirt.org/mailman/listinfo/users
>         <http://lists.ovirt.org/mailman/listinfo/users>>
>         >? ? ?>
>         >
>         >
>         >
>         >? ? ?_______________________________________________
>         >? ? ?Users mailing list
>         >? ? ?Users at ovirt.org <mailto:Users at ovirt.org>
>         <mailto:Users at ovirt.org <mailto:Users at ovirt.org>>
>         >? ? ?http://lists.ovirt.org/mailman/listinfo/users
>         <http://lists.ovirt.org/mailman/listinfo/users>
>         >? ? ?<http://lists.ovirt.org/mailman/listinfo/users
>         <http://lists.ovirt.org/mailman/listinfo/users>>
>         >
>         >
> 
> 
>         _______________________________________________
>         Users mailing list
>         Users at ovirt.org <mailto:Users at ovirt.org>
>         http://lists.ovirt.org/mailman/listinfo/users
>         <http://lists.ovirt.org/mailman/listinfo/users>
> 
> 
>     _______________________________________________
>     Gluster-users mailing list
>     Gluster-users at gluster.org <mailto:Gluster-users at
gluster.org>
>     http://lists.gluster.org/mailman/listinfo/gluster-users
>     <http://lists.gluster.org/mailman/listinfo/gluster-users>
> 
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3716 bytes
Desc: S/MIME Cryptographic Signature
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20171129/e7cd8cf7/attachment.p7s>